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FOREWORD 


As  the  Army  embarks  on  its  transformation  to  the  future,  a  variety  of  changes  will  be  required  to 
align  personnel  systems  with  future  goals  and  objectives.  In  April  2002,  the  Army  Training  and 
Leader  Development  Panel  (ATLDP)  released  the  results  of  its  survey  of  35,000 
noncommissioned  officers  (NCOs).  The  ATLDP’ s  recommendations  included  the  need  for 
regular  assessment  of  Soldiers’  technical,  tactical,  and  leadership  skills.  In  response  to  this  need, 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  issued  a  Small 
Business  Innovation  Research  (SBIR)  Phase  I  contract  entitled  “Cost-effective,  Realistic 
Measures  of  Job  Performance.”  The  aim  of  the  research  project  was  to  develop  a  methodology  to 
produce  realistic  and  cost-effective  performance  measures.  The  Phase  I  research  and 
development  efforts  have  laid  the  groundwork  for  the  development  of  prototype  military 
occupational  specialty  (MOS)  performance  measures  on  a  realism  continuum  to  determine  the 
necessary  realism  for  different  job  categories. 

ARI’s  Selection  and  Assignment  Research  Unit  conducts  research,  studies,  and  analyses  of 
individual  difference  measures  (of  aptitudes,  motivations,  and  other  attributes)  related  to 
Soldiers’  job  performance.  The  primary  goal  is  to  improve  the  Army’s  selection  and 
classification,  promotion,  and  reassignment  of  enlisted  soldiers  and  officers.  The  research 
presented  in  this  report  demonstrates  how  the  Small  Business  Innovation  Research  (SBIR) 
Program  can  support  these  objectives. 

MICHELLE  SAMS 

Technical  Director 
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A  STRATEGY  TO  PRODUCE  REALISTIC,  COST-EFFECTIVE  MEASURES  OF  JOB 
PERFORMANCE 


EXECUTIVE  SUMMARY _ _ _ 

Research  Requirement: 

For  most  MOS,  the  Army  lacks  objective  measures  to  assess  the  ability  of  Soldiers  to  perform 
the  technical  components  of  their  jobs.  Such  measures  are  needed  to  support  the  full  range  of 
Army  enlisted  human  resource  requirements,  including  recruiting,  selection,  training,  self¬ 
development,  promotion,  and  transition  assistance  for  Soldiers  exiting  the  Army.  The  objective 
of  this  effort  was  to  develop  a  methodology  to  produce  realistic  and  cost-effective  measures. 

Procedure: 

Job  Performance  Systems,  Inc.  (JPS)  and  its  subcontractor,  the  Human  Resources  Research 
Organization  (HumRRO)  began  this  effort  by  identifying  11  types  of  assessment  methods  having 
the  potential  for  use  in  this  effort.  Included  in  the  list  were  computer-based  tests  and  simulations 
designed  to  create  an  engaging,  virtual  representation  of  a  military  occupational  specialty 
(MOS). 

Participants  in  a  clustering  workshop  used  this  list,  descriptions  of  MOS,  and  other  resource 
materials,  to  identify  seven  groups  of  MOS.  Common  to  all  MOS  in  a  group  was  their  suitability 
for  assessment  using  a  specific  type  of  assessment  method(s).  The  seven  clusters  of  MOS 
provide  a  structure  for  pursuing  performance  measures  across  the  Army. 

The  JPS  team  then  developed  a  two-phased  strategy  for  collecting  job  analysis  information.  A 
preliminary  data  collection  effort  will  be  conducted  to  ensure  target  MOS  have  been  classified 
into  the  appropriate  clusters.  A  more  in-depth  analysis  follows  to  collect  the  information  needed 
to  design  the  performance  measure. 

As  part  of  the  methodology  the  team  developed  tools  to  quantify  the  level  of  realism  achieved  in 
performance  measures.  The  team  also  identified  strategies  to  reduce  the  costs  required  to 
develop  and  implement  them. 

Findings: 

No  single  assessment  method  will  be  effective  across  the  diverse  range  of  MOS  in  the  Army. 
However,  MOS  can  be  grouped  into  a  relatively  small  number  of  clusters  for  which  the  same 
type(s)  of  assessment  method(s)  can  be  used.  For  each  group  it  should  be  possible  to  create  a 
standard,  streamlined  approach  to  job  analysis,  assessment  design,  assessment  development,  and 
pilot  testing.  This  document  provides  a  structure  and  an  approach  to  achieve  this  goal. 

To  the  extent  possible,  we  sought  to  identify  assessment  methods  capable  of  being  delivered  on  a 
personal  computer.  For  some  MOS  this  approach  will  be  successful.  However,  there  are  many 
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predetermined  standards.  For  such  MOS  some  form  of  a  hands-on  measure  must  be  a 
component  of  the  assessment.  We  believe  there  are  a  number  of  viable  strategies  to  make  such 
assessments  beyond  traditional  work  sample  tests.  These  include  field-based  hands-on 
performance  tests  and  the  evaluation  of  work  products. 

Utilization  of  Findings 

For  follow-up  work  that  could  be  conducted,  we  recommend  that  a  small  sample  of  performance 
measures  be  developed  following  the  strategies  presented  in  this  document.  The  MOS  should  be 
drawn  from  the  different  clusters  identified  in  this  effort.  The  resulting  measures  should  be 
evaluated  for  their  level  of  realism.  Lessons  learned  from  Phase  II  should  be  used  to  identify 
strategies  to  standardize  and  streamline  the  process  of  developing  additional  measures  in  each 
cluster. 
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A  STRATEGY  TO  PRODUCE  REALISTIC,  COST-EFFECTIVE  MEASURES  OF  JOB 

PERFORMANCE 

I.  Introduction 


Background 

In  the  1970s,  the  U.S.  Army  established  an  ambitious  program  for  periodically  certifying  the 
technical  skills  of  enlisted  personnel.  The  Skill  Qualification  Test  (SQT)  program  included  both 
traditional  multiple-choice  and  hands-on  tests  of  proceduralized  knowledge  related  to  critical 
tasks  in  each  military  occupational  specialty  (MOS).  Without  the  aid  of  technology-based 
systems  and  with  decreasing  numbers  of  personnel  to  develop  and  maintain  the  SQT,  the 
program  was  gradually  reduced  in  scope  (e.g.,  written  tests  only,  voluntary  testing)  and 
eventually  discontinued  altogether. 

The  SQTs  were  an  important  tool  for  training,  promotion  decisions,  and  selection  and 
classification  research.  Today,  those  needs  remain  with  no  systematic  assessment  of  MOS 
technical  skills  available  to  address  them.  Indeed,  the  recent  Army  Training  and  Leader 
Development  Panel  (ATLDP)  recommended  the  Army  “develop  and  sustain  a  competency 
assessment  program  for  evaluating  Soldiers’  technical  and  tactical  proficiency  in  the  MOS  and 
leadership  skills  for  their  rank”  (ATLDP,  2002,  p.  34).  This  recommendation  has  been  approved 
as  a  required  action,  with  the  leadership  skills  element  to  be  addressed  by  tools  from  an  ongoing 
research  project,  Maximizing  21st  Century  Noncommissioned  Officer  (NCOs)  Performance 
(NC021),  sponsored  by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI).  The  products  from  this  effort  were  a  future-oriented  job  analysis  for  NCOs  and  validated 
tools  for  assessing  critical  future  knowledge,  skills,  and  attributes.  No  such  tools  currently  exist 
to  routinely  assess  technical  competence.  The  goal  of  this  project  is  to  address  this  gap. 

In  addition  to  the  concerns  of  the  ATLDP,  valid  and  cost-effective  assessments  of  technical  skills 
(and  the  job  analysis  information  required  to  develop  them)  will  support  the  full  range  of  Army 
enlisted  human  resource  requirements,  including  recruiting,  selection,  training,  self-development, 
promotion,  and  transition  assistance  for  Soldiers  exiting  the  Army.  For  example,  such  tools  will 
facilitate  the  linkage  of  Army  training  to  occupational  certification  requirements  in  the  civilian 
sector.  This  is  a  significant  initiative  being  pursued  by  the  Army  because  of  its  anticipated 
positive  impact  on  recruiting  and  transition  assistance. 

Overall  Objective 

The  overall  objective  of  this  effort  is  to  devise  effective  and  affordable  means  to  assess  the 
ability  of  Soldiers  to  perform  the  technical  requirements  of  their  MOS.  To  the  extent  possible, 
the  Army  would  like  to  assess  performance  in  a  manner  that  realistically  captures  the 
complexities  and  demands  incumbents  face  in  doing  their  jobs.  Rather  than  assess  isolated  tasks, 
the  idea  is  to  assess  performance  within  the  broader  context  of  how  each  MOS  is  actually 
performed. 
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This  research  effort  is  supported  under  the  Small  Business  Innovative  Research  (SBIR)  program. 
This  program  promotes  research  for  government  efforts  that  small  business  can  then  market  to 
the  private  sector.  Each  SBIR  is  structured  as  a  three-phase  effort.  The  first  phase  is  the 
conceptual  phase  with  products  such  as  a  detailed  methodology,  prototype,  or  conceptual 
demonstration.  In  the  second  phase,  the  research  is  conducted  over  a  two-year  period.  The  third 
phase  is  intended  for  the  adaptation  of  the  research  products  for  the  private  sector.  Following  the 
format  of  SBIR  efforts,  our  plan  is  to  pursue  performance  measures  in  a  three-phased  effort. 

•  Phase  I  -  Develop  a  methodology  for  creating  realistic  and  cost-effective  measures 

•  Phase  II  -  Demonstrate  and  evaluate  the  methodology  with  up  to  five  MOS 

•  Phase  III  -  Apply  methodology  across  Army  MOS 

Goals  for  Phase  I 

This  document  presents  the  methodology  developed  in  Phase  I.  The  methodology  was  designed 
to  achieve  the  following  goals: 

•  Produce  realistic  and  affordable  measures. 

•  Be  applicable  to  most  Army  MOS. 

•  Produce  assessments  that  are  reliable,  discriminate  between  acceptable  and  unacceptable 
performance,  and  are  practical  to  use. 

•  Support  administration  of  Soldier  assessments  irrespective  of  their  location. 

Key  Elements  to  Our  Methodology 

Our  approach  involved  the  integration  of  three  innovations.  The  first  is  conceptual,  the  second 
technological,  and  the  third  organizational. 

Assessment  Methods  linked  to  MOS  Clusters.  We  began  with  the  assumption  that  no  single  type 
of  test  will  yield  satisfactory  results  for  all  Army  MOS.  Thus,  we  identified  a  set  of  assessment 
methods  capable  of  covering  the  broad  range  of  Army  MOS.  To  gain  efficiencies  in  the  process 
of  assessment  development,  we  identified  clusters  of  MOS  needing  specific  testing  strategies. 

Computer-Based  Testing.  Our  approach  capitalizes  on  the  capabilities  of  today’s  powerful 
multimedia  personal  computers  and  high  speed  data  communications.  In  our  vision,  as  much  of 
the  testing  as  possible  will  be  conducted  on  a  computer.  For  some  jobs  we  anticipate  going  quite 
far  in  using  the  full  capabilities  of  the  computer  to  create  a  rich  and  engaging  virtual  job 
environment. 

Continuous  Process  Improvement.  The  third  innovation  is  organizational.  It  can  take  a 
substantial  effort  from  a  multidisciplinary  team  to  produce  and  maintain  realistic  and  effective 
performance  measures,  particularly  those  that  run  on  computers.  However,  the  steps  in  the 
development  and  maintenance  systems  can  be  described,  the  time  and  resources  it  takes  to 
complete  them  can  be  measured,  and  interventions  can  be  taken  to  make  the  process  more  and 
more  efficient.  A  focus  in  all  three  Phases  of  this  SBIR  will  be  to  identify  ways  to  reduce  the 
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time  and  costs  required  to  perform  job  analysis,  test  design,  development,  evaluation,  and 
maintenance  activities. 
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II.  Overview 


Our  approach  is  based  upon  clusters  of  MOS  linked  to  assessment  methods.  Common  to  all 
MOS  in  a  cluster  is  their  suitability  for  evaluation  by  a  specific  assessment  method  or 
combination  of  methods.  This  idea  is  represented  by  the  left  and  middle  boxes  in  Figure  1 .  It  is 
our  belief  that  this  approach  offers  the  best  chance  to  achieve  a  solution  that  optimizes  the 
multiple  goals  of  the  project  (i.e.,  realism,  cost-effectiveness,  validity,  and  practicality). 


MOS  Cluster 

'  MOS  Cluster  1 

1  MOS  Cluster  2 
•:  MOS  Cluster  3  - 

R'"""  . 

MOS  Cluster  N  - 

| 


Assessment  Method  Job  Analysis 


Method(s)  1 


Method(s)  2 

“|  V 

-  Method(s)  3 

if-  : 

|k- .  •'  ^  ■  /•:•'!  is 

I?;":  ••  ' 

-  Method(s)  N 

k  '  :  .r 


<-  Job  Analysis  Strategy  1 

r 

^  Job  Analysis  Strategy  2 
<—  Job  Analysis  Strategy  3 

< —  Job  Analysis  Strategy  N 


Figure  1:  Depiction  of  the  methodology. 

We  will  devise  a  common  job  analysis  approach  for  all  the  MOS  in  a  particular  cluster.  This 
idea  is  represented  by  the  box  on  the  right  side  of  the  figure. 

Our  methodology  takes  a  middle  position  between  the  extremes  of  trying  to  impose  a  single  type 
of  assessment  approach  on  all  Army  MOS  and  allowing  any  evaluation  option  for  every  MOS. 
With  our  methodology  we  identify,  up  front,  a  limited  and  viable  set  of  assessment  strategies. 
These  strategies  are  selected  on  the  basis  of  their  potential  to  achieve  the  goals  of  the  project  and 
their  applicability  to  a  broad  range  of  Army  MOS. 

Linking  MOS  clusters  to  assessment  methods  offers  the  possibility  of  devising  a  standard 
approach  to  producing  the  assessments  for  the  MOS  in  each  cluster.  This  means  the  same 
approach  can  be  applied  to  performing  the  job  analysis,  test  design,  and  test  development 
activities.  We  anticipate  having  a  proven,  streamlined  process  after  just  a  few  iterations  of 
producing  assessments  for  MOS  in  a  particular  cluster. 

With  some  initial  experience  in  developing  such  measures,  it  should  also  be  possible  to  generate 
accurate  estimates  of  the  time  and  costs  needed  to  create  performance  measures  for  use  across 
the  entire  Army. 
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III.  Assessment  Methods 


Initial  Set  of  Methods 

Members  of  the  JPS  team  have  had  extensive  practical  experience  in  developing  assessment 
measures  used  in  the  Army  and  the  commercial  sector.  We  started  the  process  of  identifying 
methods  by  simply  asking  team  members  to  nominate  those  approaches  they  felt  had  potential  to 
achieve  the  goals  of  the  project.  Table  1  presents  the  results  of  this  step. 


Table  7:  Initial  List  of  Assessment  Methods 


Method 

1. 

On-the-job  monitoring/assessment  (embedded  in  relevant  systems) 

2. 

Supervisor  ratings 

3. 

Expert  evaluation  of  actual  work  products 

4. 

High  fidelity  hands-on  work  sample  tests  (process  and/or  product  scoring)  that  closely  model 
task  requirements 

5. 

Lower  fidelity  hands-on  work  sample  tests  (process  and/or  product  scoring)  requiring 
skipping  or  talking  through  aspects  of  performance  (e.g.,  due  to  time/safety  considerations) 

6. 

High  fidelity  simulation  using  Army  training  equipment/systems 

7. 

High  fidelity  computer-based  simulation  (programming,  equipment,  and  props  to  closely 
match  the  real  thing) 

8. 

Medium  fidelity  computer-based  simulation  (programming,  visuals,  and  low  cost  PC 
attachments  to  obtain  a  reasonable  approximation  of  the  real  thing) 

9. 

Low  fidelity  computer-based  simulation  (programming  and  visuals  that  get  the  message 
across,  but  does  not  closely  match  the  real  thing) 

Proceduralized  knowledge  multiple-choice  test  (liberal  use  of  audio/visuals) 

um 

Proceduralized  knowledge  multiple-choice  test  (mostly  text-based) 

IEW 

Declarative  knowledge  multiple-choice  test  (liberal  use  of  audio/visuals) 

|13^_ 

Declarative  knowledge  multiple-choice  test  (mostly  text-based) 

We  arranged  the  list  so  that  the  methods  appearing  further  up  tend  to  offer  higher  levels  of 
realism.  Ordering  the  list  in  this  way  illustrated  to  us  one  of  the  challenges  of  this  project. 

Those  types  that  inherently  offer  greater  realism  tend  to  also  be  substantially  more  costly  to 
design,  build,  administer,  maintain,  and  upgrade. 

As  mentioned  in  the  introduction  of  this  document,  our  vision  is  that  as  much  testing  as  possible 
be  delivered  via  computer.  Hosting  tests  on  this  platform  supports  a  number  of  strategies  we  can 
pursue  to  reduce  costs  and  promote  realism.  It  can  also  help  us  facilitate  test  administration, 
improve  accuracy  of  scoring,  promote  accessibility,  and  support  test  maintenance.  Finally, 
hosting  tests  on  this  platform  and  recording  scores  at  a  central  server  can  facilitate  the  conduct  of 
reliability  and  validity  studies.  However,  several  assessment  types  located  toward  the  top  of  the 
list  can  not  be  easily  administered  via  computer.  These  include  on-the-job 
monitoring/assessment,  expert  evaluation  of  actual  work  products,  and  most  work  sample  tests. 

Several  types  of  multiple-choice  tests  appear  toward  the  bottom  of  the  list.  Given  the  focus  on 
realism,  it  may  seem  out  of  place  to  have  included  them  on  our  list.  Our  thought,  however,  was 
that  with  some  creativity,  it  will  be  useful  at  times  to  include  this  technique  as  a  separate 
component  to  a  test  battery  or  to  integrate  it  into  another  performance  assessment  technique. 
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When  administered  on  a  computer,  questions  can  be  presented  using  audiovisual  stimuli  and 
sequenced  in  an  order  that  matches  how  tasks  are  actually  completed.  Adding  in  such  items  as 
electronic  versions  of  equipment  manuals,  simulated  test  equipment,  and  other  simulated  features 
of  the  job  moves  the  test  in  the  direction  of  a  virtual  job. 

As  an  aside,  we  had  a  number  of  exchanges  among  ourselves  regarding  what  made  one  type  of 
test  distinct  from  another.  For  example  it  became  clear  to  us  that  one  could  blur  the  distinctions 
between  multiple-choice  tests  and  simulations.  We  sharpened  the  distinctions  by  defining 
“multiple-choice”  as  any  close-ended  response  format  item,  including  ranking  and  “drag  and 
drop”  problems.  We  distinguished  them  from  simulations  in  that  they  do  not  allow  the  examinee 
to  interact  with  the  test  stimulus. 

Changes  to  the  List 

We  made  several  modifications  to  the  list  early  in  the  course  of  the  Phase  I  effort.  Some  of  the 
changes  reflected  concerns  about  the  feasibility  of  using  particular  testing  methods  in  the  Army. 
Some  changes  were  made  to  shorten  the  list  and  make  it  more  manageable  for  our  purposes. 
Finally,  some  changes  resulted  from  issues  that  surfaced  when  we  considered  how  we  might 
apply  the  test  methods  to  a  sample  of  MOS. 

For  example,  an  early  decision  was  to  remove  supervisor  ratings  from  the  list  (method  2  in  Table 
1).  We  felt  the  inherent  subjectivity  of  supervisor  ratings  and  their  tendency  to  become  inflated 
when  used  for  decision-making  purposes  outweighed  any  advantages  this  test  method  might 
offer. 

We  also  made  an  early  decision  to  remove  text-based  declarative  knowledge  multiple-choice 
exams  (method  13)  from  the  list.  We  simply  felt  that  this  testing  method  was  inconsistent  with 
the  goal  of  realism  in  testing.  The  better  choice  was  to  pursue  multiple-choice  testing  in  which 
the  items  were  made  to  be  as  performance-oriented  as  possible  (i.e.,  assess  proceduralized 
knowledge).  This  is  as  opposed  to  questions  that  are  posed  out  of  any  real-world  context  and 
require  simple  recall  of  facts  or  procedures  (i.e.,  purely  declarative  knowledge). 

Another  decision  was  to  combine  methods  6  and  7  (high  fidelity  testing  using  existing  Army 
equipment  and  high  fidelity  simulation  developed  on  a  computer).  These  methods  differ  only  in 
whether  or  not  there  is  an  existing  training  simulator  that  might  be  used  for  testing.  There  was 
no  value  to  us  in  keeping  method  6  separate  as  it  was  beyond  our  scope  to  investigate  for  each 
MOS  the  existence  of  suitable  simulators  and  their  availability  for  testing  purposes. 

We  also  decided  to  modify  our  list  of  the  remaining  multiple-choice  testing  methods  (methods 
10,  11,  and  12).  We  simplified  the  entries  by  dropping  any  distinctions  between  testing 
knowledge  and  procedures  and  instead  focusing  upon  the  difference  between  using  relatively 
inexpensive  visuals  and  more  elaborate  multimedia  presentation  techniques. 

We  then  added  a  type  of  multiple-choice  test  in  which  the  items  unfold  naturally  within  the 
context  of  job-relevant  scenarios  (presented  via  computerized  simulations,  video,  and/or  text). 
Such  a  test  might  consist  of  a  series  of  small  scenarios,  focusing  on  different  aspects  of  the  job. 
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Alternatively,  it  could  be  a  single  large  scenario  that  captures  the  essence  of  all  key  job  areas.  In 
this  option,  it  would  also  be  possible  to  incorporate  some  semblance  of  contextual  factors  such  as 
interactions  with  others,  interruptions,  and  time  pressures. 

Finally,  we  felt  it  useful  to  add  situational  judgment  tests  to  our  list.  This  issue  arose  while 
considering  how  we  might  assess  MOS  such  as  Military  Police  in  which  effective  performance  at 
times  requires  the  Soldier  to  exercise  significant  levels  of  judgment  in  ambiguous  or  novel 
situations. 

Our  revised  list  of  test  methods  appears  in  Table  2.  The  next  chapter  explains  how  we  used  this 
list  to  help  establish  MOS  clusters. 


Table  2:  Revised  Assessment  Methods 


Method 

1 

On-the-job  monitoring/assessment  (embedded  in  relevant  systems) 

2 

Expert  evaluation  of  actual  work  products 

3 

High  fidelity  hands-on  work  sample  tests  (process  and/or  product  scoring)  that  closely  model 
task  requirements 

4 

Lower  fidelity  hands-on  work  sample  tests  (process  and/or  product  scoring)  requiring 
skipping  or  talking  through  aspects  of  performance  (e.g.,  due  to  time/safety  considerations) 

High  fidelity  computer-based  simulation  (programming,  equipment,  and  props  to  closely 
match  the  real  thing)  that  may  make  use  of  Army  training  simulators 

6 

Medium  fidelity  computer-based  simulation  (programming,  visuals,  and  low  cost  PC 
attachments  to  obtain  a  reasonable  approximation  of  the  real  thing) 

7 

Low  fidelity  computer-based  simulation  (programming  and  visuals  that  get  the  message 
across,  but  does  not  closely  match  the  real  thing) 

8 

Multiple-choice  “simulation”  (items  organized  into  one  or  more  cohesive  scenarios  using 
audio/video  clips  and  some  computerized  simulation;  despite  their  inherent  interdependence, 
items  are  designed  to  be  as  independent  as  possible) 

9 

Multiple-choice  situational  judgment  test  (with  or  without  audio/video  clips) 

10 

Multiple-choice  test  (using  audio/video  clips  and  some  computerized  simulation) 

Multiple-choice  test  (using  simple  visuals  such  as  photos,  figures,  and  graphs) 
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IV.  Identification  of  MOS  Clusters 

We  scheduled  a  workshop  on  June  1 7,  2003  to  generate  the  MOS  clusters.  In  advance  of  the 
workshop,  we  prepared  a  set  of  materials  for  use  by  participants. 

Workshop  Materials 

The  first  item  in  the  set  was  the  list  of  revised  assessment  methods  that  appear  in  Table  2.  The 
second  item  was  a  list  of  1 1  MOS  characteristics  we  felt  should  underlie  decisions  about  which 
test  methods  were  most  appropriate  for  jobs.  This  list  appears  in  Appendix  A.  As  an  example, 
one  characteristic  was  the  extent  to  which  the  procedural  knowledge  and  skill  in  manipulating 
objects  was  required  in  job  performance.  Jobs  high  on  this  characteristic  would  likely  entail 
significant  amounts  of  physical  action.  This  characteristic  in  turn  might  suggest  a  cluster  of 
MOS  in  which  the  ideal  testing  method  would  be  some  sort  of  hands-on  work  sample  test. 

As  another  example,  Appendix  A  contains  a  characteristic  we  called  systems  thinking.  This  we 
defined  in  terms  of  needing  to  manage  processes  characterized  by  a  number  of  fairly  complex, 
interrelated  elements.  This  characteristic  might  suggest  another  cluster  in  which  simulations 
would  be  the  best  type  of  test  method  to  use. 

A  third  item  for  participants’  use  was  a  set  of  50  descriptions  of  Army  MOS  derived  from  U.S. 
Army  Pamphlet  611-21  (Department  of  the  Army,  1999).  The  list  covers  all  MOS  that  exist  at 
Skill  Level  1  (or  may  exist  at  this  level  in  the  future).  The  descriptions  were  typically  half  a  page 
to  a  full  page  in  length.  Each  was  headed  with  a  job  number  and  a  job  title.  This  was  followed  by 
a  brief  description  of  the  major  duties.  Next  was  a  sentence  or  two  describing  the  requirements  at 
skill  levels  1  and  2.  The  descriptions  also  indicated  whether  the  MOS  was  open  to  women  and 
the  level  of  physical  demands.  As  an  example,  a  description  for  Cannon  Crewmember  appears  in 
Appendix  B. 

The  50  descriptions  were  actually  a  sample  of  a  larger  set  of  roughly  175  descriptions  that  had 
been  previously  generated  as  part  of  the  Select21  project.  The  order  of  the  list  was  randomized 
so  that  the  50  we  used  in  the  workshop  were  reasonably  representative  of  the  entire  set. 

The  fourth  item  was  a  list  of  all  the  potential  job  analysis  methods  we  might  use  to  help  design 
the  tests  we  would  construct  in  Phases  II  and  III  of  this  effort.  We  identified  seven  methods: 

•  Direct  observation 

•  Analysis  of  existing  materials 

•  Interviews 

•  Subject  matter  experts 

•  Cross-job  questionnaires 

•  Within-job  questionnaires 

•  Critical  incidents/scenarios 

Definitions  of  these  methods  appear  in  Appendix  C. 
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Workshop  Pilot  Test 

On  June  10,  2003  three  project  staff  members  and  a  representative  from  ARI  met  to  review  the 
workshop  materials  and  to  try  out  the  clustering  process  with  four  MOS  descriptions.  This 
activity  led  us  to  add  clarifying  descriptions  to  some  of  our  materials.  It  also  emphasized  the 
value  of  having  participants  among  us  who  were  knowledgeable  enough  about  the  MOS  to 
answer  questions  as  we  attempted  to  generate  clusters. 

Workshop  Process 

Six  individuals  participated  in  the  clustering  workshop1.  We  began  the  event  by  reviewing  the 
goals  of  the  project  and  the  advantages  we  hoped  to  achieve  with  them  in  completing  Phases  II 
and  III  of  the  SBIR  project.  We  explained  that  a  good  clustering  solution  would  be  one  with  a 
manageable  number  of  groups  (i.e.,  three  to  ten  clusters)  and  be  based  on  aspects  of  MOS  that 
related  directly  to  the  choice  of  test  methods. 

We  provided  the  resource  materials  to  each  participant  and  offered  examples  of  how  one  could 
take  a  description  of  an  MOS,  apply  the  other  resource  material  to  it,  and  place  the  MOS  into  a 
possible  cluster.  Each  participant  was  provided  the  same  randomized  set  of  MOS  descriptions. 
Given  time  constraints,  we  restricted  the  number  of  MOS  to  50  rather  than  use  the  entire  set  of 
175.  Participants  had  approximately  2.5  hours  to  work  independently  in  constructing  their 
cluster  solution.  The  focus  of  this  clustering  exercise  was  to  put  MOS  together  that  shared 
characteristics  (e.g.,  procedural  knowledge  and  skill  for  manipulating  objects,  level  of  cognitive 
complexity)  suggesting  a  common  subset  of  preferred  assessment  methods. 

Participants  described  the  clusters  they  each  identified.  The  group  then  arrived  at  a  consensus 
decision  on  a  set  of  seven  clusters.  For  each  cluster,  the  group  assigned  a  type  of  test  or 
combination  of  tests  to  it.  To  better  understand  the  meaning  and  utility  of  the  clusters,  the  group 
sorted  all  50  MOS  into  them.  The  following  week,  two  members  of  the  JPS  team  finished 
sorting  the  remaining  MOS  into  the  same  seven  clusters. 

MOS  Cluster  Results 

Appendix  D  shows  how  each  MOS  was  sorted.  We  caution  the  reader  that  these  results  are 
preliminary  and  MOS  will  likely  be  moved  among  clusters  as  we  gain  more  information  about 
them  in  future  SBIR  phases. 

Table  3  contains  summary  information  about  the  clusters.  As  the  second  column  shows,  we 
found  the  number  of  MOS  per  cluster  varies  greatly.  The  smallest,  Cluster  6  could  have  just  a 
single  MOS.  The  largest,  Cluster  4,  likely  has  well  over  a  1 00. 


1  The  judges  included  John  Campbell,  Roy  Campbell,  Peter  Greenston,  Deirdre  Knapp,  Doug  Rosenthal,  and 
Christopher  Sager. 
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Table  3:  MOS  Clusters 


Cluster 

Estim.  # 
MOS  in 
Cluster 

Example  of 
MOS  in 
Cluster 

Assessment 

Environment 

Assessment  Items 

1 

15 

Personnel 

Admin. 

Specialist 

Virtual  Job 

Independent  multiple-choice 
items 

2 

28 

Human 

Intelligence 

Collector 

Virtual  Job 

Multiple-choice  items  presented 
as  part  of  job  scenarios 

3 

5 

Broadcast 

Journalist 

Real  Job 

Evaluation  of  final  work 
products 

4 

110 

Cannon 

Crewmember 

Virtual  Job 

Multiple-choice  items  (presented 
independently  or  as  part  of 
scenarios) 

Real  Job 

•  Hands  on  demonstration 

•  Evaluation  of  final  work 
products 

•  Embedded  on-the-job 
monitoring 

•  Historical  records 

5 

4 

Common 

Ground 

Station 

Operator 

Virtual  Job 

Multiple-choice  items 

Virtual  Job 

Low  to  medium  fidelity 
interactive  simulation 

6 

1 

Air  Traffic 
Controller 

Virtual  Job 

Multiple-choice  items 

Virtual  Job 

High  fidelity  interactive 
simulation 

7 

7 

Military 

Police 

Virtual  Job 

Multiple-choice  items 

Virtual  Job 

Situational  judgment  test  items 

Real  Job 

•  Hands  on  demonstration 

•  Evaluation  of  final  work 
products 

•  Embedded  on-the-job 
monitoring 

•  Historical  records 

The  assessment  environment  (column  4)  concerns  whether  performance  should  be  assessed  with 
a  PC,  with  some  type  of  on-the-job  activity,  or  both.  The  term  Virtual  Job  is  used  to  convey  the 
vision  of  using  a  PC  to  display  some  type  of  engaging,  realistic  testing  environment,  even  if  the 
test  questions  are  simply  multiple-choice  items.  The  Virtual  Job  might  show  photographs,  maps, 
diagrams,  equipment  manuals,  and  other  job  relevant  documents.  It  might  use  video  clips.  It 
might  even  provide  simulated  test  equipment. 
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The  last  column  in  the  table  indicates  the  kind  of  evaluation  items  we  believe  are  appropriate. 

For  virtual  environments  they  range  from  multiple-choice  items  to  the  outcomes  that  occur  in 
interacting  with  a  high  fidelity  simulation.  The  last  column  in  Table  3  provides  information 
about  the  specific  testing  methods  we  recommend  for  each  cluster. 

For  many  MOS,  we  decided  some  type  of  hands-on  test  was  needed.  A  traditional  work  sample 
test  would  be  an  example  of  such  a  test.  This  was  the  case  for  MOS  sorted  into  Clusters  3, 4,  and 
7  (denoted  by  the  term  Real  Job  in  column  4).  Many  jobs  in  these  clusters  appear  to  require  that 
a  trained  evaluator  observe  the  person  performing  a  physical  action  to  a  predetermined  standard. 

Those  MOS  in  Cluster  3  can  be  assessed  solely  by  means  of  a  hands-on  test.  Those  in  Cluster  4 
would  be  assessed  using  a  hands-on  test  in  combination  with  a  PC  delivered  multiple-choice  test. 
Those  in  Cluster  7  might  need  a  situational  judgment  test  as  part  of  the  testing  battery. 

It  is  not  reasonable,  however,  to  expect  the  Army  to  fund  the  development  and  implementation 
of  traditional  work  sample  tests  considering  the  large  number  of  MOS  in  these  clusters 
(particularly  Cluster  4).  We  therefore  identified  four  alternative  approaches  to  collecting  on-the- 
job  information.  These  are  hands-on-demonstration  tests,  evaluation  of  final  work  products, 
embedded  on-the-job  monitoring,  and  historical  records. 

Alternatives  to  Traditional  Work  Sample  Tests 

Hands-On-Demonstration  Test 

The  concept  here  is  that  the  Soldier  and  supervisor  are  given  much  of  the  responsibility  to 
conduct  the  testing  event  themselves.  During  this  event  Soldiers  perform  job  tasks  (as  part  of 
their  normal  job)  and  are  evaluated/certified  by  a  supervisor.  The  Soldier  and  supervisor  would 
be  given  a  set  window  of  weeks  or  months  in  which  to  complete  this  assignment.  Perhaps  the 
final  decision  the  supervisor  makes  is  dichotomous.  That  is,  a  go  (task  done  right)/no  go  (task 
not  done  right  -  still  needs  to  learn  the  job  better  and  be  re-evaluated  later). 

Our  role  would  be  to  structure  the  process  to  ensure  the  soldier  and  supervisor  perform  it 
correctly.  We  would  provide  training  to  the  supervisor  (over  the  web)  on  performance 
assessment.  The  training  would  cover  how  to  use  the  task  checklist  we  send  (over  the  web),  how 
to  work  in  the  testing  process  as  part  of  their  other  duties,  how  to  evaluate  if  task  performance  is 
acceptable  (determine  if  the  soldier  achieved  the  standard),  how  to  give  appropriate  feedback, 
and  how  to  enter  the  results  in  a  data  base. 

In  combination  with  some  type  of  computer  delivered  knowledge  test  taken  by  the  solider,  this 
might  be  a  very  workable  solution  for  many  MOS  in  Cluster  4.  It  would  be  in  line  with  the 
emphasis  the  Army  now  puts  on  teaching  level  2  and  beyond  Soldiers  how  to  train  and  evaluate 
training  results.  As  a  side  benefit,  it  might  make  supervisors  throughout  the  Army  significantly 
more  skilled  in  determining  the  proficiency  of  their  Soldiers.  This  would  complement  their 
traditional  focus  on  training. 
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Evaluation  of  Final  Work  Products 


There  appear  to  be  some  jobs  in  which  the  incumbent  can  be  asked  to  submit  a  sample  of  final 
work  products  for  review.  This  may,  in  fact,  be  the  only  strategy  that  makes  sense  for  jobs  such 
as  multimedia  artist,  journalist,  or  translator.  The  products  could  be  evaluated  by  the  supervisor 
or  a  panel  of  experts. 

Embedded  On-the-Job  Monitoring 

Given  the  increasing  use  of  technology  in  MOS,  there  may  be  significant  opportunities  for 
technology  to  play  an  increasing  role  in  capturing  job  performance. 

Historical  Records 


Finally,  for  a  few  jobs,  it  may  make  sense  to  review  historical  data  on  a  Soldier’s  performance  in 
a  job.  Consider  the  job  of  a  parachute  rigger.  We  understand  that  any  incident  resulting  from 
problems  in  the  inspection  or  packing  of  a  parachute  can  be  tracked  back  to  the  person  who 
prepared  it. 
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V.  Job  Analysis  Strategies 


Overview 

We  envision  a  2-stage  job  analysis  process.  In  the  first  stage,  a  “preliminary”  job  analysis  will  be 
conducted  to  confirm  a  priori  decisions  regarding  the  job  cluster  a  given  MOS  has  been  assigned 
to,  and  by  implication,  the  measurement  method(s)  that  will  be  used  to  form  the  performance 
assessment.  The  second  stage  of  the  job  analysis  will  provide  the  information  needed  to  design 
the  assessments.  It  will  provide  the  basis  for  content  specifications,  but  not  necessarily  all  the 
content  (e.g.,  video  clips)  required  to  support  measure  development. 

During  the  course  of  the  2-stage  job  analysis  work,  there  will  be  some  information  we  anticipate 
collecting  for  all  jobs: 

(1)  A  delineation  of  the  major  job  activities  (e.g.,  tasks,  task  categories,  roles)  that  can  be 
used  to  ensure  the  assessments)  cover  relevant  material  and  are  suitably 
comprehensive  in  scope. 

(2)  An  understanding  of  contextual  factors,  such  as  the  degree  of  interaction  with  others, 
the  need  for  multi-tasking,  and  so  forth  that  can  be  used  to  infuse  the  assessment 
experience  with  realism. 

(3)  An  understanding  of  how  the  job  is  performed  and  to  what  standard  (e.g.,  as  depicted 
in  Soldier’s  Manuals),  which  becomes  the  basis  for  determining  the  “right  answers” 
and  assessing  performance. 


Issues 

The  performance  assessments  developed  for  the  Army  are  expected  to  be  realistic  and  cost- 
effective.  An  important  strategy  for  incorporating  realism  into  an  assessment  involves  having  a 
holistic  view  of  job  requirements.  Major  drivers  on  the  cost-effectiveness  issue  include  having 
assessments  with  a  reasonable  life  cycle  (i.e.,  do  not  require  constant  changes  to  keep  them  up- 
to-date)  and  that  do  not  involve  separate  measures  for  Soldiers  with  different  assignments.  Our 
proposed  methods  for  collecting  preliminary  and  full  job  analysis  information  are  designed  to 
address,  inasmuch  as  possible,  several  factors  that  complicate  performing  job  analysis  work  that 
supports  these  goals.  Those  factors  include: 

■  Unit  differences  in  job  requirements  and  equipment. 

■  Swiftly  evolving  job  requirements  and  equipment. 

■  Traditional  focus  on  detailed  job  tasks. 

■  Differences  in  field,  school,  and  doctrinal  perspectives. 

For  example,  the  Army’s  traditional  focus  on  a  very  fine  grain  task  analysis  -  which  might  be 
more  appropriately  termed  learning  objectives  -  could  lead  an  analyst  away  from  a  more  holistic 
approach  to  performance  assessment.  Detailed  tasks  can  also  change  fairly  quickly  whereas 
broader  job  descriptors  are  likely  to  be  more  stable.  At  the  same  time,  an  approach  is  needed  that 
provides  sufficient  depth  to  answer  the  many  questions  that  arise  in  measure  development. 
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Another  issue  is  the  utility  of  available  sources  of  job  information.  Soldier’s  Manuals  that  detail 
task  requirements  for  each  MOS  are  in  various  stages  of  currency.  The  Army  Occupational 
Survey  Program  provides  another  obvious  source  of  job  analysis  information,  but  the  adequacy 
of  the  information  varies  considerably  across  MOS.  For  this  reason,  the  same  basic  job  analysis 
procedure  applied  to  two  different  MOS  may  vary  considerably  in  resource  requirements  due  to 
differences  in  the  availability  of  up-to-date,  quality  information  from  existing  sources. 

Preliminary  Job  Analysis 

As  mentioned  above,  the  primary  goal  of  the  preliminary  job  analysis  is  to  collect  sufficient 
information  about  selected  MOS  to  determine  whether  each  has  been  assigned  to  the  correct  job 
cluster.  Confirmation  of  this  assignment  will  allow  us  to  (a)  make  a  final  determination  of  the 
types  of  measures  to  develop  for  each  target  MOS  and  (b)  design  the  detailed  job  analysis 
approach  for  each  MOS  so  that  it  will  be  efficient  and  support  the  development  of  the  necessary 
assessments. 

The  procedure,  materials,  and  results  from  the  job  clustering  exercise  serve  as  a  good  starting 
point  for  this  stage.  For  example,  the  list  of  MOS  characteristics  (Appendix  A)  outlines  much  of 
the  information  that  is  important  for  identifying  the  appropriate  combination  of  measures  for 
assessing  performance  in  an  MOS.  The  clustering  exercise  also  revealed  that  more  information  is 
needed  to  confirm  our  initial  assignments  of  MOS  to  clusters.  One  issue  was  that  participants  felt 
that  the  MOS  descriptions  used  in  the  exercise  did  not  have  enough  information  to  allow  them  to 
make  confident  assignments.  Additionally,  it  became  apparent  that  some  information  not 
formally  considered  during  the  exercise  might  affect  cluster  assignments.  For  example,  if  an 
MOS  is  very  small  or  about  to  be  collapsed  into  another,  it  might  not  be  cost  effective  to  develop 
an  expensive  simulation  for  that  MOS. 

Proposed  Procedure 

With  these  goals  and  this  recent  experience  in  mind,  we  propose  the  following  procedure.  The 
primary  product  will  be  a  completed  Preliminary  Job  Analysis  Form  for  each  target  occupation. 
The  information  contained  in  these  forms  will  help  to  confirm  the  cluster  assignment  and  design 
the  rest  of  the  job  analysis  approach  for  each  MOS.  Appendix  E  contains  the  following  support 
materials: 

■  Instructions  for  Job  Analysts:  Completing  the  Preliminary  Job  Analysis  Form 

■  Example  Preliminary  Job  Analysis  Form  (1  IB  Infantryman) 

■  Taxonomy  of  Potential  Major  Work  Activities 

■  Definitions  of  MOS  Characteristics 

The  Preliminary  Job  Analysis  Form  includes  three  major  sections.  The  first  is  titled  Major  MOS- 
Specific  Activities.  The  idea  here,  as  described  in  the  instructions,  is  to  use  a  broad  taxonomy  of 
work  activities  to  identify  and  describe  the  most  important  elements  of  the  MOS  in  terms  of 
MOS-specific  work  activities  (e.g.,  Entering  buildings  during  an  urban  operations  and  Zeroing  a 
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weapon)2.  These  work  activities  will  serve  at  least  two  purposes:  (a)  they  will  support  inferences 
about  the  required  knowledges  and  skills  and  (b)  they  will  offer  early  suggestions  about 
assessment  content. 

MOS  Characteristics  is  the  second  section.  It  includes  characteristics  that  project  staff  identified 
as  relevant  to  determining  appropriate  measurement  methods  during  preparation  for  the 
clustering  exercise.  MOS  Characteristics  includes  four  subsections;  the  first  is  Determinants  of 
Performance.  The  determinants  of  performance  are  derived  primarily  from  the  job  performance 
modeling  literature  (e.g.,  Campbell,  McCloy,  Oppler,  &  Sager,  1993;  Campbell  &  Kuncel, 

2001).  The  theory  states  that  two  important  determinants  of  job  performance  are  Declarative 
Knowledge  (DK)  and  Procedural  Knowledge  and  Skill  (PKS).  DK  is  defined  as  knowledge  of 
facts  and  things,  while  PKS  is  defined  as  the  successful  combination  of  “knowing  what  do”  with 
“knowing  how  to  do  it.”  PKS  is  especially  apparent  when  mastery  depends  on  practice  (e.g., 
word  processing  and  interacting  with  the  public).  The  extent  to  which  performance  on  a  job  is 
determined  by  DK  or  a  particular  PKS  can  have  a  substantial  influence  on  the  selection  of 
measurement  methods.  For  example,  if  DK  is  the  primary  determinant  of  performance,  some 
kind  of  multiple-choice  test  is  likely  to  work.  However,  if  the  PKS  Manipulating  Objects  is  an 
important  determinant  of  performance,  then  including  a  hands-on  measure  that  can  test  that  skill 
may  be  necessary. 

The  remaining  MOS  characteristics  are  described  in  Definitions  of  MOS  Characteristics.  They 
include  Cognitive  Complexity,  Work  Context,  and  Tools,  Equipment  and  Systems.  Cognitive 
complexity  addresses  issues  relating  to  the  level  of  cognitive  demand  the  MOS  places  on  the 
individual  Soldier.  For  example,  Air  Traffic  Controller  is  an  occupation  that  places  a  substantial 
cognitive  demand  on  the  incumbent;  whereas,  the  level  of  cognitive  demand  placed  on  a  Ticket 
Taker  is  somewhat  less.  Work  Context  addresses  elements  of  the  work  environment  that  we  think 
are  relevant  to  the  identification  of  appropriate  measurement  methods.  The  items  were  derived 
primarily  from  the  Work  Context  portion  of  the  Occupational  Information  Network  (0*NET) 
taxonomy  (Strong,  Jeanneret,  McPhail,  Blakely,  &  D’egidio,  1999).  Tools,  Equipment,  and 
Systems  will  be  a  list  of  the  major  tools,  pieces  of  equipment,  and  systems  that  are  used  by  all  or 
most  of  the  Skill  Level  1  Soldiers  in  the  target  MOS. 

The  last  part  of  the  form  is  Practical  Considerations.  Here  we  include  additional  information 
that  is  relevant  to  developing  performance  measures  for  this  MOS.  Practical  considerations  will 
include  the  likely  future  of  the  MOS  and  an  estimate  of  the  number  of  Skill  Level  1  Soldiers 
currently  in  the  MOS.  This  kind  of  information  is  important  because  it  might  impact  decisions 
about  the  practicality  or  cost-effectiveness  of  developing  certain  kinds  of  assessments  for 
particular  MOS. 

The  Instructions  for  Job  Analysts:  Completing  the  Preliminary  Job  Analysis  Form  describe  the 
process  we  propose  for  completing  this  form.  It  includes  a  review  of  existing  materials  (e.g.,  task 
lists  and  Soldier  manuals)  to  develop  an  initial  draft,  an  interview  (that  might  include  direct 


2  The  Taxonomy  of  Potential  Major  Work  Activities  in  Appendix  G  is  derived  from  the  0*NET  generalized  work 
activities  (Jeanneret,  Borman,  Kubisiak,  &  Hanson,  1999).  An  alternative  we  are  considering  is  based  on  the  Army 
Synthetic  Validation  project  (Wise,  Peterson,  Hoffman,  Campbell,  &  Arabian,  1991). 
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observation),  and  a  series  of  project  staff  reviews  that  result  in  a  final  and  complete  version  of 
the  form. 

Confirmation  of  Cluster  Assignments 

Individual  project  staff  members  and  sponsor  representatives  will  then  review  the  information  in 
the  preliminary  job  analysis  forms.  This  review  will  be  followed  by  a  consensus  meeting  to 
confirm  and/or  adjust  the  measurement  approach  cluster  assignments  for  target  MOS.  Important 
considerations  will  include: 

■  If  an  MOS  is  dominated  by  DK  and  PKS  is  not  that  important,  some  form  of  multiple- 
choice  test  is  likely  to  be  sufficient. 

■  If  an  MOS  includes  a  lot  of  manipulating  objects  (i.e.,  a  PKS)  that  require  well-developed 
physical  and/or  psychomotor  skills  to  perform  important  activities,  some  kind  of  hands- 
on,  work  sample,  or  imbedded  on  the  job  monitoring  will  be  necessary. 

■  MOS  in  which  performance  depends  substantially  on  PKSs  like  interacting  with  others, 
using  information  resources,  reading,  and/or  writing  will  need  an  assessment  approach 
that  addresses  these  skills.  Possibilities  include  computer  simulations,  situational 
judgment  tests  (SJTs),  and  evaluations  of  final  work  products. 

■  MOS  that  involve  a  substantial  amount  of  judgment/problem  solving  in  situations  that  are 
not  easily  addressed  by  specific  knowledge(s)  or  skill(s)  are  likely  to  require  a  set  of 
assessments  that  include  an  SJT  or  some  kind  of  simulation  that  can  present  these 
situations  with  sufficient  fidelity. 

■  If  an  MOS  requires  a  great  deal  of  systems  thinking,  its  performance  measures  will 
probably  need  to  include  a  simulation  that  requires  the  Soldier  to  deal  with  a  number  of 
interdependent  pieces  of  information,  decisions  and/or  required  outputs. 

■  Information  about  work  context,  tools,  equipment,  and  systems  will  help  to  further 
identify  characteristics  assessments  for  individual  MOS  should  address.  Some  examples 
include: 

-  If  a  substantial  portion  of  the  work  for  an  MOS  takes  place  via  a  computer/visual 
display,  a  computerized  simulation  is  likely  to  have  high  validity  and  realism. 

-  If  dealing  well  with  time  pressure  and  interruptions  is  important  to  job 
performance  in  an  MOS,  this  can  be  simulated  by  some  of  the  proposed  measures. 

-  If  skill  in  the  use  of  a  particular  tool  or  piece  of  equipment  (e.g.,  a  personal 
weapon)  is  very  important,  a  hands-on  test  covering  it  may  be  necessary. 

Full  Job  Analysis 

The  full  job  analysis  is  designed  to  collect  enough  information  to  design  the  performance 
assessments)  identified  for  the  target  MOS.  As  shown  in  Figure  1,  the  job  analysis  method  will 
be  linked  to  the  requirements  of  the  test  method(s)  associated  with  each  job  cluster.  Particularly 
given  that  we  expect  the  clusters  themselves  to  evolve  as  we  move  into  Phase  II,  we  will  not  try 
to  delineate  full  job  analysis  procedures  in  great  detail.  Rather  our  goal  here  is  to  provide  an  idea 
of  how  the  procedures  are  likely  to  vary  across  clusters. 
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Methods  Common  to  All  Job  Clusters 


A  broad  list  of  job  analysis  methods  is  provided  on  page  1 1  and  detailed  in  Appendix  C.  We 
recommend  that  some  of  these  methods  be  used  for  all  MOS,  regardless  of  job  cluster. 
Specifically,  all  job  analyses  will  include  analysis  of  existing  materials,  at  least  one  SME 
workshop,  and  a  field  survey  of  incumbents  and/or  supervisors.  (Table  4  shows  our  initial 
recommendations  about  the  job  analysis  methods  applicable  for  each  job  cluster.) 


Table  4.  Job  Analysis  Methods  by  Job  Clusters 


Cluster/ 

Method 

1 

2 

3 

H 

5 

6 

7 

Direct  observation 

X 

a 

Analysis  of  existing  docs 

X 

X 

a 

X 

a 

SME  Interviews 

a 

SME  Workshops 

E9 

□ 

a 

Q 

□ 

Cross-job  surveys 

Within  job  surveys  (broad) 

Q 

Within  job  surveys  (detailed) 

X 

a 

X 

X 

X 

Critical  incidents/scenarios 

X 

The  first  SME  workshop  may  be  conducted  in  conjunction  with  completion  of  the  Preliminary 
Job  Analysis  Form.  Subsequent  workshops  may  be  needed  to  provide  sufficient  feedback  on  the 
list  of  major  work  activities  to  use  in  a  survey.  What  will  be  different  across  clusters  is  the  level 
of  detail  at  which  major  work  activities  must  be  defined  and  prioritized  through  SME  workshops 
and  a  field  survey. 

Common  to  all  job  analyses  will  be  work  activities/job  descriptors  organized  into  the  generic 
work  activities  discussed  in  the  preliminary  job  analysis  section.  Our  goal  is  to  include  job- 
specific  information  as  needed  but  try  for  efficiencies  of  process  by  attempting  to  stay  within  the 
organizing  structure  of  the  modified  0*NET  work  activities  (Peterson,  Mumford,  Borman, 
Jeanneret,  &  Fleishman,  1999).  In  all  cases,  job  descriptors  within  the  major  work  activities  will 
be  less  detailed  than  the  tasks  the  Army  has  traditionally  used  for  training  (which  might  be  more 
appropriately  called  learning  objectives).  For  clusters  requiring  multiple-choice  or  hands-on 
tests,  the  job  descriptors  will  need  to  be  more  detailed  (e.g.,  broadly- worded  tasks)  than  the  other 
clusters  (major  job  functions  or  task  categories). 

SME  workshops.  To  the  extent  possible,  we  will  use  existing  materials  to  craft  the  work 
activities  that  will  be  embedded  in  the  field  survey.  SME  workshops  will  be  used  as  needed  to 
finalize  the  list.  SMEs  will  be  identified  through  the  applicable  proponent  schools.  In  Phase  II, 
we  want  to  try  conducting  at  least  some  SME  workshops  in  a  virtual  environment  (e.g.,  using 
web-supported  teleconferences)  as  a  cost-savings  measure. 

Field  survey.  We  propose  using  automated  surveys  to  collect  data  for  prioritizing  work 
activities  for  measurement.  We  recommend  using  the  AUTOGEN  software  ARI’s  Occupational 
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Analysis  Office  (OAO)  has  developed  to  collect  the  required  data  (U.S.  Army  Training  and 
Doctrine  Command,  2004).  Unlike  the  current  use  of  AUTOGEN,  which  was  designed  so  that 
proponents  could  conduct  job  analysis  work  without  additional  support  from  ARI,  we  would 
require  that  professional  job  analysts  familiar  with  project  requirements  participate  in  the  entire 
process.  The  sampling  plan  for  each  survey  would  be  worked  out  with  the  MOS  proponent. 

Methods  for  Selected  Job  Clusters 

Direct  observation.  Clusters  #4  and  #7  involve  hands-on  tests  that  will  likely  require 
direct  observation  of  work  performance  to  design,  particularly  given  the  goal  of  high  fidelity 
testing  that  provides  a  realistic  context  for  the  testing  situation. 

SME  interviews.  Some  assessment  methods,  particularly  those  requiring  computerized 
simulations,  will  require  focused  time  with  individual  SMEs  to  collect  the  detailed  “walk¬ 
through”  information  needed  to  design  the  simulations. 

Critical  incidents/scenarios.  Development  of  a  situational  judgment  test  (S  JT)  will 
require  job  analysis  information  that  (a)  identifies  the  critical  behavioral  areas  that  should  be 
covered  on  the  test  and  (b)  serves  as  the  raw  material  for  developing  questions.  Therefore, 
critical  incident/scenario  workshops  with  job  incumbents  will  be  required  for  those  MOS  in 
Cluster  #7.  Again,  we  recommend  trying  to  conduct  some  of  this  workshop  activity  using  virtual 
meetings  supported  by  technology. 

Job  Analysis  Updates 

If  it  is  to  support  up-to-date  assessments  without  undue  resource  burdens,  an  important  feature  of 
the  job  analysis  procedures  will  be  to  incorporate  routine  mechanisms  for  updating  information 
(Knapp,  Morath,  Quartetti,  &  Ramos,  1997).  This  contrasts  with  the  traditional  expectation  that  a 
big  job  analysis  is  conducted  every  3-5  years.  At  a  minimum,  this  would  entail  re-administration 
of  the  field  survey  to  a  relatively  small  sample  of  respondents  every  12  months  or  so. 

Introduction  of  new  equipment  or  other  known  MOS  changes  could  lead  to  adjustments  in  this 
schedule.  Such  changes  might  also  be  addressed  by  conducting  an  interim  survey  that  focuses 
only  on  those  aspects  of  the  job  expected  to  be  affected  by  the  changes  in  progress. 
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VI.  Process  Improvement  Strategy 


It  will  require  a  significant  level  of  resources  to  produce  and  implement  performance  measures 
throughout  the  Army.  However,  the  steps  to  the  process  can  be  documented,  time  and  cost 
metrics  can  be  developed,  and  strategies  can  be  applied  to  achieve  continuous  improvements  in 
efficiency. 

The  work  required  to  build,  implement,  and  maintain  performance  measures  in  the  Army  can  be 
divided  into  seven  main  activities.  Each  activity  represents  a  target  for  continuous  improvement. 


Table  5:  Targets  for  Process  Improvement  Activities 


Activity 

1. 

Preliminary  Job  Analysis 

Hi 

Full  Job  Analysis 

a 

Design 

EM 

Development 

Pilot  Test 

6. 

Implementation 

7. 

Support  and  Maintenance 

Current  Strategies  to  Achieve  Cost-Effective  Measures 

Our  goal  in  Phase  I  was  to  devise  a  methodology  in  which  cost  effectiveness  was  an  inherent 
feature.  The  identification  of  clusters  of  MOS  tied  to  test  methods  represents  our  core  strategy  to 
achieve  this  result,  while  simultaneously  achieving  the  requirements  for  realism  and  validity. 

The  previous  chapter  outlined  a  strategy  to  create  a  standard,  efficient  process  for  collecting  the 
job  information  we  will  need  to  ensure  MOS  have  been  assigned  to  the  correct  cluster  and  to 
design  the  measures.  This  will  clearly  be  a  fruitful  area  for  us  to  revise  and  streamline  over  time. 
Our  team  has  found  that  roughly  two-thirds  of  the  labor  hours  we  spend  to  produce  job 
simulations  are  devoted  to  the  job  analysis  and  design  activities. 

One  strategy  we  have  already  developed  to  make  the  design  and  development  process  more 
efficient  is  to  produce  a  written  document  up  front  that  answers  many  of  the  questions  we  need 
to  know  to  achieve  a  successful  product.  This  document  also  helps  reduce  misunderstandings 
between  our  team  and  the  client. 

Appendix  F  contains  a  customized  template  for  producing  such  a  document  in  our  Phase  II  work. 
The  result  will  be  a  separate  design  guide  we  will  create  for  each  measure  we  develop.  It  will  be 
completed  as  an  initial  step  in  the  design  phase  of  the  work.  We  will  provide  a  copy  to  ARI  and 
the  MOS  proponent  and  indicate  those  sections  for  which  we  request  their  approval  prior  to  our 
starting  development  work. 

Section  XI  of  the  template  requires  the  project  manager  to  establish  and  describe  specific  process 
improvement  goals.  These  may  include  new  tools,  procedures,  or  templates  that  will  be 
developed  and  evaluated  to  make  the  process  of  developing  another  test  like  it  (i.e.,  an  MOS  in 
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the  same  cluster)  more  standardized  and  streamlined.  Our  team  has  a  formal  mechanism  to 
identify  ideas  to  improve  our  work.  We  have  designed  what  we  call  a  “post-mortem”  meeting. 
This  is  both  a  celebration  of  the  completion  of  a  project  and  a  serious  discussion  among  all 
contributors  about  what  went  well  and  what  we  might  do  better  next  time. 

For  each  formal  process  improvement  initiative  we  undertake  in  Phases  II  and  III,  we  will  collect 
quantitative  and  qualitative  data  to  evaluate  our  success.  A  key  metric  we  will  track  is  the 
number  of  hours  we  expend  in  each  project  activity.  We  have  a  web-based  time  keeping  system 
hosted  on  a  server.  For  any  project  we  can  list  the  specific  tasks  (and  subtasks).  Contributors 
can  then  indicate  what  specific  areas  of  the  project  they  performed  as  they  enter  their  hours.  The 
system  makes  it  very  easy  to  calculate  total  hours  spent  on  any  portion  of  a  project.  We  will  use 
this  system  to  determine  the  effectiveness  of  our  improvement  initiatives  in  reducing  labor  hour 
requirements. 

Future  Effort  to  Achieve  Additional  Efficiencies 

The  puipose  of  Phase  II  will  be  to  conduct  a  feasibility  study  of  our  methodology  with  a  small 
sample  of  MOS.  Thus,  by  design  we  will  be  particularly  focused  on  devising  and  improving 
procedures  associated  with  activities  1  through  5  in  Table  5.  In  Phase  III,  our  focus  expands  to 
include  full  implementation,  support,  and  life  cycle  maintenance  (i.e.,  activities  6  and  7). 

One  of  the  more  intractable  bottlenecks  we  experience  with  our  clients  is  having  access  to  their 
SMEs.  Among  the  solutions  we  have  considered  at  JPS  are  ways  to  use  the  internet  to  better 
communicate  and  exchange  information.  We  have  already  noted  our  interest  in  pursuing  this 
medium  to  more  efficiency  perform  job  analysis  tasks.  We  believe  this  is  just  one  of  several 
opportunities  that  exist  to  develop  methods  of  making  better  use  of  SMEs.  We  anticipate  this 
being  one  of  several  areas  we  will  explore  for  process  improvement  initiatives  in  Phases  II  and 
III  that  can  lead  to  savings  in  both  costs  and  time  required  to  produce  and  maintain  performance 
measures. 


20 


VII.  Meaning  and  Measurement  of  Realism 


We  began  the  Phase  I  effort  with  a  review  of  the  scientific  literature  on  realism  in  performance 
measures.  Our  goal  was  to  identify  a  definition  of  the  concept  and  develop  an  operational 
measure  of  it  (Rosenthal,  2003).  However,  our  literature  review  suggested  that  realism  in  test 
performance  has  received  little  critical  analysis  and  there  is  no  generally  accepted  definition  of  it. 

For  our  purposes  we  defined  realism  as  a  combination  of  the  test’s  physical  and  psychological 
fidelity  to  the  job.  Of  paramount  importance  to  achieving  physical  fidelity  is  that: 

•  There  is  a  close  match  between  the  stimuli  and  response  options  afforded  on  the  test  and 
in  the  job 

•  Important  contextual  features  that  facilitate  or  inhibit  performance  on  the  job  are 
represented  in  the  test 

Of  paramount  importance  to  achieving  psychological  fidelity  is  that  the  content  of  the  test  (i.e., 
the  challenges,  situations,  and  scenarios)  are  relevant  and  well  constructed.  There  are  other 
features  to  a  test  that  can  add  to  its  overall  level  of  realism.  Whether  their  impact  works  though 
improving  the  physical  or  psychological  fidelity  is  not  clear.  These  include  the  pace/intensity  of 
the  action  and  the  extent  to  which  the  events  cover  significant  portions  of  the  job  (e.g.,  complete 
tasks,  duties,  or  activities). 

For  computer-administered  tests,  we  developed  rating  scales  that  can  be  used  to  produce  a 
quantitative  measure  of  the  extent  to  which  they  are  realistic  (See  Appendix  G).  We  will  use 
these  scales  to  help  evaluate  our  success  in  achieving  satisfactory  levels  of  realism  in  tests  we 
create  in  Phases  II  and  III. 
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VIII.  Conclusions  and  Recommendations 


The  results  of  this  effort  support  the  conclusion  that  MOS  can  be  grouped  into  a  relatively  small 
number  of  clusters  for  which  the  same  type(s)  of  assessment  method  can  be  used.  As  a  result,  it 
appears  likely  that  a  standard,  streamlined  approach  to  job  analysis,  test  design,  test 
development,  and  pilot  testing  can  be  developed  for  each  cluster. 

The  workshop  experience  convinces  us  that  some  of  the  MOS  we  assigned  to  clusters  need  to  be 
examined  more  closely  in  the  future  and  possibly  reclassified.  Our  team  lacked  the  in-depth 
knowledge  of  each  MOS  needed  to  ensure  all  classifications  were  correct.  However,  our 
methodology,  which  includes  a  preliminary  job  analysis,  should  correct  this  problem. 

While  we  identified  seven  groups  of  MOS,  the  number  of  MOS  in  each  group  varied  widely.  In 
fact,  the  two  clusters  requiring  interactive  simulations  appear  to  have  very  few  MOS  in  them. 

The  Army  may  therefore  decide  not  to  make  them  a  priority  for  evaluation  in  Phase  II. 

On  the  other  hand,  one  cluster  contained  well  over  a  hundred  MOS.  In  Phase  II  we  will 
determine  if  this  cluster  deserves  to  be  further  divided.  Our  current  thinking  is  that  this  is  likely. 
The  subdivision  may  be  among  different  strategies  to  collect  hands-on  data.  It  may  also  be 
divided  among  those  MOS  having  useful  existing  job  information  (e.g.,  up-to-date  Soldier’s 
Manuals)  and  those  lacking  any  useful  job  information. 

To  the  extent  possible,  we  hoped  to  assess  performance  using  methods  delivered  on  a  personal 
computer.  For  some  MOS  this  approach  will  be  successful.  However,  there  are  many  MOS  in 
which  it  is  critical  that  the  Soldier  perform  a  series  of  physical  actions  to  predetermined 
standards.  For  such  MOS  some  form  of  on-the-job  assessment  must  be  a  component  of  the 
testing.  We  believe  there  are  a  number  of  viable  strategies  to  make  such  assessments  beyond 
traditional  work  sample  tests.  These  include  hands-on  demonstration  tests  and  the  evaluation  of 
work  products. 

The  focus  of  our  efforts  was  upon  MOS  performance  at  Skill  Level  1 .  It  seems  logical  to  assume 
that  the  need  for  using  cognitive  skills  will  generally  increase  while  the  need  for  physical  skills 
will  generally  decrease  at  higher  and  higher  skill  levels.  Thus,  at  the  next  level  (i.e.,  Skill  Level 
2),  the  need  for  hands-on  assessment  may  decrease  and  relatively  more  Soldiers  may  be 
effectively  assessed  using  just  a  computer. 

For  Phase  II  we  recommend  that  a  small  sample  of  performance  measures  be  developed 
following  the  strategies  presented  in  this  document.  The  MOS  should  be  drawn  from  the 
different  clusters  identified  in  this  effort.  The  resulting  measures  should  be  evaluated  for  their 
level  of  realism.  Lessons  learned  from  Phase  II  should  be  used  to  identify  strategies  to 
standardize  and  streamline  the  process  of  developing  additional  measures  in  each  cluster. 


22 


REFERENCES 


ATLDP  (2002).  The  Army  Training  and  Leader  Development  Panel  Report  (NCO), 
http://www.army.mil/features/atldpnco/NCO_STUDY_REPORT.pdf. 

Campbell,  J.P.,  &  Kuncel,  N.R.  (2001).  Individual  and  team  training.  In  N.  Anderson,  D.  One,  &  H. 
Sinangil  (Eds.),  Handbook  of  Industrial,  Work,  and  Organizational  Psychology:  Personnel 
Psychology  (Volume  1 ).  Thousand  Oaks,  CA:  Sage  Publications,  Inc. 

Campbell,  J.P.,  McCloy,  R.A.,  Oppler,  S.H.,  &  Sager,  C.E.  (1993).  A  theory  of  performance.  In  N. 
Schmitt,  &  W.C.  Borman  (Eds.),  Personnel  selection  in  organizations  (pp.  35-70).  San 
Francisco:  Jossey-Bass. 

Department  of  the  Army.  (1999).  Pamphlet  611-21  Personnel  Selection  and  Classification, 
Military  Occupational  Classification  and  Structure.  Washington,  DC:  Headquarters, 
Author. 

Jeanneret,  P.R.,  Borman,  W.C.,  Kubisiak,  U.C.,  and  Hanson,  M.A.  (1999).  Generalized  work 
activities.  In  N.G.  Peterson,  M.D.  Mumford,  W.C.  Borman,  P.R.  Jeanneret,  &  E.A. 
Fleishman  (Eds.),  An  occupational  information  system  for  the  21st  century:  The  development 
ofO*NET( pp.  105-125).  Washington,  DC:  American  Psychological  Association. 

Knapp,  D.J.,  Morath,  R.,  Quartetti,  D.A.,  &  Ramos,  R.A.  (1997).  A  strategic  approach  to  the 
analysis  of  air  traffic  controller  jobs.  Fairfax,  VA:  Caliber  Associates,  Inc. 

Peterson,  N.G.,  Mumford,  M.D.,  Borman,  W.C.,  Jeanneret,  P.R.,  &  Fleishman,  E.A.  (Eds.) 
(1999).  An  occupational  information  system  for  the  21st  century:  The  development  of 
0*NET.  Washington,  DC:  American  Psychological  Association. 

Rosenthal,  D.  B.  (2003).  The  meaning  and  measurement  of  realism  in  tests  of  job  performance. 
Alexandria,  VA:  Job  Performance  Systems,  Inc. 

Strong,  M.H.,  Jeanneret,  P.R.,  McPhail,  S.M.,  Blakely,  B.R.,  &  D’egidio,  E.L.  (1999).  Work 
context:  Taxonomy  and  measurement  of  the  work  environment.  In  N.G.  Peterson,  M.D. 
Mumford,  W.C.  Borman,  P.R.  Jeanneret,  &  E.A.  Fleishman  (Eds.),  An  occupational 
information  system  for  the  21st  century:  The  development  of  0*NET  (pp.  127-145). 
Washington,  DC:  American  Psychological  Association. 

U.S.  Army  Training  and  Doctrine  Command.  (27  Jan  2004).  Memorandum,  Mandatory  Use  of 
AUTOGEN.  FT  Monroe,  VA:  Author. 

Wise,  L.L.,  Peterson,  N.G.,  Hoffman,  R.G.,  Campbell,  J.P.,  &  Arabian,  J.M.  (1991).  Army 
Synthetic  Validity  Project:  Report  of  phase  III  results-Vol.  I.  (Technical  Report  922). 
Alexandria,  VA:  U.S.  Army  Research  Institute  for  Behavioral  and  Social  Sciences. 


23 


24 


Appendix  A 
MOS  Characteristics 
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1.  Declarative  Knowledge  -  knowledge  of  facts  and  things;  including  lists  of  steps  in  a 
procedure  (i.e.,  knowing  what  to  do).  Test  items/scenarios  addressing  declarative 
knowledge  focus  on  asking  incumbents  how  they  do  their  work. 

Note:  In  very  practical  terms  this  means  knowledge  (including  some  knowledge 
application)  that  could  be  assessed  with  a  multiple-choice  test,  including  test 
items  with  graphical,  audio,  and/or  visual  stems  and/or  response  options. 

2.  Procedural  Knowledge  &  Skill  (PK&S)/Manipulating  Objects3  -  (e.g.,  assembling  a 
weapon,  operating  a  cash-register,  aiming  a  large  weapon) 

3.  PK&S/Interacting  with  others  -  effective  interaction  with  others  (e.g.,  customers, 
patients,  team  members); 

-  Interpersonal  Skills 

-  Listening 

-  Speaking 

4.  PK&S/Using  information  resources  -  looking  stuff  up  in  a  manual,  data  base,  or 
intra/intemet 

5.  PK&S/Reading 

6.  PK&S/Writing  [Reading  and  writing  refer  specifically  to  activities  required  by  the  job, 
not  to  these  skills  in  the  general  context.] 

7.  Cognitive  Complexity  -  complex  decision-making,  judgment,  and/or  processing  a  lot  of 
information  at  once 

8.  Systems  Thinking  -  understanding  a  “system”  with  a  number  of  interrelated  elements  that 
affect  each  other.  Examples  of  jobs  high  on  this  characteristic  include,  Air  Traffic 
Controller,  Water  Treatment  Specialist,  and  Dispatcher.  An  MOS  completely  lacking  in 
this  characteristic  would  be  one  in  which  very  few  of  its  tasks  are  dependent  on  each 
other  (i.e.,  a  number  of  discrete  tasks  where  performance  on  one  tasks  is  not  affected  by 
performance  on  the  others). 

9.  Time  pressure/decision  speed  -  performing  work  or  making  decisions  quickly 

10.  Computer/Visual  Display  -  The  MOS  takes  place  primarily  on  a  computer  or  some  other 
type  of  visual  display 

11.  Judgment  -  work  in  the  MOS  is  characterized  by  a  fair  number  of  situations  where  the 
Soldier  needs  to  exercise  judgment  regarding  problems  that  are  not  easily  addressed  by  a 
specific  piece  of  knowledge  or  skill  [Here  we  are  thinking  about  Critical  Incidents  and/or 
SJT  scenarios.  Situations  in  which  judgment  might  be  important,  even  for  Skill  Level  1 
Soldiers:  (a)  there  is  a  regulation,  rule,  or  procedure,  but  recognizing  that  it  is  relevant  or 
necessary  to  follow  is  difficult  for  some;  (b)  the  manuals  don’t  cover  this  (a  lot  of 
interpersonal,  team,  and  leadership  stuff  might  fall  in  this  category);  and  (c)  different, 
potentially  contradictory,  policies  seem  to  be  relevant  to  the  situation.] 


3  The  following  definition  applies  of  all  of  the  PK&  S  characteristics.  Procedural  Knowledge  &  Skill  -  the 
successful  combination  of  “knowing  what  to  do”  with  “knowing  how  to  do  it;”  this  is  especially 
prominent  when  mastery  depends  on  practice  (e.g.,  word  processing  or  interacting  with  customers).  Test 
items/scenarios  addressing  procedural  knowledge  focus  on  requiring  incumbents  to  simulate  doing  part  of 
their  work. 
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Appendix  B 

Sample  MOS  Description 
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Job  Number:  3 

Job  Title:  Cannon  Crewmember 

Major  Duty:  Supervises  or  serves  as  a  member  of  field  artillery  weapon  systems. 

Skill  Level  1:  Establishes  and  maintains  radio  and  wire  communications.  Maintains,  prepares,  and 
loads  ammunition  for  firing.  Stores,  maintains,  and  distributes  ammunition  to  using 
units  as  a  member  of  battery  or  battalion  ammunition  section.  Operates  and  performs 
operator  maintenance  on  prime  movers,  self-propelled  howitzers,  ammunition  vehicles, 
and  other  unit  vehicles.  Performs  crew  maintenance  and  participates  in  organizational 
maintenance  of  weapons  and  related  equipment. 

Skill  Level  2:  Supervises  handling,  transportation,  accountability,  and  distribution  of  ammunition. 

Assists  section  chief  in  supervision  of  howitzer  operations,  maintenance,  and  training. 
Lays  weapon  for  direction,  conducts  bore  sighting  and  basic  periodic  tests.  Supervises 
the  operation,  loading,  and  maintenance  of  the  Field  Artillery  Ammunition  Support 
Vehicle 

Gender:  closed  to  women 

Physical  Demands:  very  heavy 

PhysP  PhysU  PhysL  PhysH  PhysE  PhysS 

Physical  Profile:  2  2  2  2  2  1 

Clearance  Level:  none  Skill  Level  >  1 :  No 
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Appendix  C 

Potential  Job  Analysis  Strategies 
for  use  with  MOS  Clusters 
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1.  Direct  Observation  -  job  analysts  observe  incumbents  performing  work.  Information  can 
be  collected  in  a  narrative  format  or  in  some  more  structured  way  (e.g.,  checklist,  rating 
scales,  worksheet,  etc.) 

2.  Analysis  of  Existing  Materials  -  job  analysts  extract  information  from  existing  job  related 
artifacts  including: 

o  Standard  Operating  Procedures  (SOPs) 
o  Manuals 
o  Equipment/tools 

o  Training  texts,  workbooks,  and  test  items 
o  Previous  job  analysis  results 
o  Published  performance  or  certification  standards 

3.  Interviews  -  job  analysts  conduct  interviews  of  incumbents  and/or  supervisors.  Interview 
can  cover  topics  such  as  (a)  tasks,  (b)  knowledges,  skills,  and  attributes  (KSAs),  and  (c) 
the  relative  importance  of  these  characteristics.  This  category  includes  “protocol 
analysis”  where  an  incumbent  is  asked  to  provide  a  narrative  of  physical  and  mental 
behaviors  as  they  are  performing  (or  pretending  to  perform)  an  element  of  the  job. 

4.  Subject  Matter  Expert  (SME)  Workshops  -  this  method  collects  the  same  types  of 
information  as  interviews  with  the  exception  that  average  or  consensus  ratings  and 
judgments  can  be  collected. 

5.  Cross-job  Questionnaires  -  generally  incumbents,  supervisors,  or  analysts  rate  the 
importance,  level  required,  and/or  frequency  of  particular  task,  activities,  or  KSAs 
relative  to  the  performance  of  the  job.  Cross-job  questionnaires  include  characteristics 
(e.g.,  activities,  abilities,  or  skills)  that  could  be  thought  of  as  applying  to  multiple  jobs  or 
any  job  (e.g.,  the  0*NET  skills,  abilities,  and  work  styles). 

6.  Within-job  Questionnaires  -  this  method  collects  the  same  types  of  information  as  cross¬ 
job  questionnaires  except  that  items  tend  to  be  job  specific.  For  example,  the  tasks  on  a 
Plumber  questionnaire  would  all  be  things  that  Plumbers  probably  do  and  an  office 
manager  would  endorse  almost  none  of  these  tasks 

7.  Critical  Incidents/Scenarios  -  the  collection  of  a  series  of  anecdotes  about  on  the  job 
behaviors  that  tend  to  be  examples  of  particularly  good  or  poor  performance.  They  are 
generally  collected  from  incumbents  and/or  supervisors.  Critical  incidents  have  three 
essential  parts:  (a)  a  description  of  the  situation  that  led  up  to  the  incident,  (b)  a 
description  of  what  the  individual  did  in  response,  and  (c)  a  description  of  the 
consequences  of  the  response.  The  word  scenario  in  this  method’s  title  refers  to  the 
notion  that  some  variations  involve  the  collection  of  richer,  more  detailed,  descriptions  of 
events  than  the  traditional  “critical  incident”  technique. 
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Appendix  D 
MOS  Clusters 


i  i 
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Cluster  1: 

Multiple-Choice  Questions 

ID 

Number 

MOS  Title 

4 

Water  Treatment  Specialist 

6 

Bridge  Crewmember 

25 

Animal  Care  Specialist 

38 

Food  Service  Operations 

39 

Signals  Collection/Identification  Analyst 

57 

Imagery  Analyst 

93 

Medical  Supply  Specialist 

98 

Communications  Locator/Interceptor 

122 

Signals  Intelligence  Analyst  (SIGINT  Analyst) 

135 

Ground  Surveillance  Systems  (GSS  Operator) 

127 

Veterinary  Food  Inspection  Specialist 

131 

Mortuary  Affairs  Specialist 

137 

Psychological  Operations  Specialist 

141 

Hospital  Food  Service  Specialist 

159 

Personnel  Administration  Specialist 

32 


Cluster  2:  Multiple-Choice  Questions  Presented  as  Part  of  Scenarios 


ID 

Number 

MOS  Title 

5 

Patient  Administration  Specialist 

24 

Multiple  Launch  Rocket  System  Repairer 

42 

Human  Intelligence  Collector  (HUMINT  Collector) 

43 

PATROIT  Fire  Control  Enhanced  Operator/Maintainer 

46 

Supply  Specialist 

47 

Administrative  Specialist 

56 

Counterintelligence  Agent 

75 

Aviation  Operations  Specialist 

78 

Finance  Specialist 

80 

Crytologic  Linguist 

82 

Petroleum  Laboratory  Specialist 

92 

Military  Intelligence  Systems  Maintainer/Operator 

96 

Automated  Logistical  Specialist 

KESIH 

Personnel  Services  Specialist 

WSMBiM 

Personnel  Information  System  Management  Specialist 

105 

Telecommunications  Terminal  Device  Repairer 

110 

Accounting  Specialist 

112 

Multi-channel  Transmission  Systems  Operator-Maintainer 

113 

Fire  Support  Specialist 

123 

Transportation  Management  Coordinator 

128 

Satellite  Communication  Systems  Operator-Maintainer 

138 

Cannon  Fire  Direction  Specialist 

143 

Paralegal  Specialist 

145 

Pharmacy  Specialist 

148 

Preventive  Medicine  Specialist 

156 

Radar  Repairer 

160 

Information  Systems  Operator-Analyst 

161 

Civil  Affairs  Specialist  (Reserve  Components) 
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Cluster  3:  Evaluations  of  Final  Work  Products 


ID  Number 

MOS  Title 

7 

Journalist 

36 

Combat  Documentation/Production  Specialist 

34 

Broadcast  Journalist 

116 

Multimedia  Illustrator 

175 

Translator/Interpreter  (Reserve  Components) 
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Cluster  4:  Multiple-Choice  Questions  Presented  Alone  or  as  Part  of  Scenarios 
and  Evaluation  of  Actual  Performance _ 


ID 

Number 

MOS  Title 

1 

Heavy- Wheel  Vehicle  Mechanic 

2 

Self-Propelled  Field  Artillery  System  Mechanic 

3 

Cannon  Crewmember 

8 

Parachute  Rigger 

9 

Fire  Control  System  Repairer 

10 

AVENGER  System  Repairer 

12 

Microwave  Systems  Operator-Maintainer 

13 

Medical  Laboratory  Specialist 

14 

Explosive  Ordnance  Disposal  (EOD)  Specialist 

15 

Plumber 

16 

Aircraft  Powertrain  Repairer 

17 

Indirect  Fire  Infantryman 

19 

Cargo  Specialist 

20 

Healthcare  Specialist 

21 

AH-64  Attack  Helicopter  Repairer 

22 

Dental  Specialist 

23 

Combat  Engineer 

26 

Aircraft  Pneudraulics  Repairer 

28 

Petroleum  Supply  Specialist 

29 

Network  Switching  Systems  Operator-Maintainer 

30 

Light- Wheel  Vehicle  Mechanic 

33 

Avionic  Communications  Equipment  Repairer 

35 

Topographic  Surveyor 

40 

Avionic  Mechanic 

41 

Track  Vehicle  Mechanic 

44 

Ml  Armor  Crewman 

45 

Land  Combat  Electronic  Missile  System  Repairer 

48 

Multiple  Launch  Rocket  System  Crewmember 

50 

Infantryman 

52 

Operating  Room  Specialist 

53 

UH-60  Helicopter  Repairer 

54 

MLRS  Operations/Fire  Direction  Specialist 

55 

Optical  Laboratory  Specialist 

58 

Integrated  Family  of  Test  Equipment  Operator  &  Maintainer 

59 

Small  Arms/ Artillery  Repairer 

60 

Tactical  Automated  Fire  Control  Systems  Specialist 

61 

Crane  Operator 

62 

Utilities  Equipment  Repairer 

63 

Interior  Electrician 

64 

PATRIOT  Launching  Station  Enhanced  Operator/Maintainer 
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Medical  Equipment  Repairer  _ _ 


Cable  Systems  Installer-Maintainer  _ 


General  Construction  Equipment  Operator  _ 


Concrete  and  Asphalt  Equipment  Operator _ 


Quartermaster  &  Chemical  Equipment  Repairer _ _ 


Power-Generation  Equipment  Repairer _ 


Special  Purpose  Equipment  Repairer  _ 


Wheel  Vehicle  Repairer  _ 


Air  Defense  Command,  Control,  Communications,  Computers  & 
Intelligence  Tactical  Operations  Center  Enhanced  Operator  Maintainer 
Visual  Information  Equipment  Operator-Maintainer  _ 


Air  Traffic  Control  Equipment  Repairer  _ 


Self-Propelled  Field  Artillery  Turrret  Mechanic  _ 


Radio  Operator-Maintainer  _ 


Motor  Transport  Operator 


Field  Artillery  Meteorological  Crewmember  _ 


OH-58D  Helicopter  Repairer _ _ 


Prime  Power  Production  Specialist  _ 


Avionic  Radar  Repairer  _ 


Tactical  Unmanned  Aerial  Vehicle  (TUAV)  Operator 


AVENGER  Crewmember  _ 


Technical  Engineering  Specialist  _ 


Watercraft  Operator 


Aircraft  Powerplant  Repairer _ 


Cavalry  Scout _ 

Watercraft  Engineer _ _ 


OH-58D  Armament/Electrical/Avionics  Systems  Repairer _ 


Automatic  Test  Equipment  Operator  &  Maintainer  _ 


Metal  Worker  _ 


Telecommunications  Operator-Maintainer 


Radio  &  Communications  Security  (COMSEC)  Repairer _ 

Test,  Measurement,  and  Diagnostic  Equipment  Maintenance  Support 
Specialist  _ _ _ _ 


Field  Artillery  Surveyor 


Machinist  _ 


Carpentry  &  Masonry  Specialist  _ 


Bradley  Linebacker  Crewmember  _ 


AH-64  Armament/Electrical  Systems  Repairer  _ 


Topographic  Analyst _ 


Aircraft  Electrician  _ 


Fuel  &  Electrical  Systems  Repairer  _ 


Aircraft  Structural  Repairer _ _ 


Signal  Support  Systems  Specialist  _ 


Wire  Systems  Equipment  Repairer  _ 


Lithographer 


36 


Laundry  &  Textile  Specialist _ _ 


Track  Vehicle  Repairer  _ 


Heavy  Construction  Equipment  Operator _ 

CH-47  (or  perhaps  medium)  Helicopter  Repairer _ 


Radiology  Specialist _ 

AH-64D  Armament/Electrical/ Avionics  Systems  Repairer 


Diver 

Construction  Equipment  Repairer 

UH-1  Helicopter  Repairer 

Chemical  Operations  Specialist 

Ammunitions  Specialist  _ _ 


155  Armament  Repairer _ _ _ 


Special  Electronic  Devices  Repairer  _ _ _ 


Field  Artillery  Firefinder  Radar  Operator  _ 


162  Transmission  &  Distribution  Specialist  (Reserve  Components) _ 

163  _ Observations/Scout  Helicopter  Repairer _ 

164  _ AH-1  Attack  Helicopter  Repairer _ _ _ 

165  Man  Portable  Air  Defense  System  (MANPADS)  Crewmember  (Reserve) 

166  CHAPARRAL  &  REDEYE  Repairer  _ _ 

167  _ Land  Combat  Support  System  Test  Specialist _ _ 

168  _ Turbine  Engine  Driven  Generator  Repairer _ 

169  _ Quarrying  Specialist _ _ 

170  Utility  Airplane  Repairer  (Reserve  Components) _ 

17 1  Aircraft  Armament/Missile  Systems  Repairer _ _ 

172  Railway  Equipment  Repairer  (Reserve  Components) _ 

173  Railway  Section  Repairer  (Reserve  Components) _ 

174  Railway  Operations  Crewmember  (Reserve  Components) _ 

Note.  Options  for  evaluation  of  actual  performance  are  hands  on  demonstration,  evaluation  of  final  work 
products,  embedded  on-the-job  monitoring,  or  historical  records. 
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Cluster  5: 

Low/Medium  Fidelity  Interactive  Simulation  and  Multiple-Choice  Questions 


ID 

Number 

MOS  Title 

27 

Field  Artillery  Tactical  Data  Systems  Specialist 

31 

Common  Ground  Station  (CGS)  Operator 

32 

Electronic  Intelligence  Interceptor/Analyst 

37 

Intelligence  Analyst 

38 


Cluster  6:  High  Fidelity  Interactive  Simulation  and  Multiple-Choice 
Questions _ 


ID 

Number 

MOS  Title 

51 

Air  Traffic  Control  Operator 
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Cluster  7:  Multiple-Choice  Questions,  Situational  Judgment  Test,  and 


ID 

Number 

MOS  Title 

20 

Healthcare  Specialist 

49 

Military  Police 

85 

Criminal  Investigations  Special  Agent 

90 

Chaplain  Assistant 

120 

Mental  Health  Specialist 

130 

Firefighter 

147 

Corrections  Specialist 

products,  embedded  on-the-job  monitoring,  or  historical  records. 


40 


Appendix  E 

Materials  for  Performing  a  Preliminary  Job  Analysis 
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Instructions  for  Job  Analysts: 

Completing  the  Preliminary  Job  Analysis  Form 

Goal 

The  primary  goal  is  to  collect  sufficient  job  analysis  information  about  selected  MOS  to 
determine  whether  each  of  these  MOS  has  been  assigned  to  the  correct  MOS/measurement 
approach  job  cluster.  Confirmation  of  this  assignment  will  allow  us  to  (a)  make  a  final 
determination  of  the  types  of  assessments  to  develop  for  each  MOS  and  (b)  design  the 
detailed  job  analysis  approach  for  each  MOS  so  that  it  will  be  efficient  and  support  the 
development  of  the  necessary  assessments. 

Your  Task 

Your  task  is  to  collect  the  information  necessary  to  achieve  this  goal  by  (a)  reviewing 
existing  information  (e.g.,  Soldier  manuals  and  task  lists),  (b)  conducting  interviews  and 
direct  observations  of  work,  and  (c)  completing  the  Preliminary  Job  Analysis  Form  for  your 
assigned  MOS. 

Step  1:  Review  Preliminary  Job  Analysis  Support  Materials 

Phase  1  Report  -  This  document  will  help  you  understand  how  the  preliminary  job  analysis 
fits  in  the  context  of  the  broader  project. 

Example  Preliminary  Job  Analysis  Form  -  This  is  an  example  of  the  form  you  will 
eventually  complete  for  your  assigned  MOS. 

Taxonomy  of  Potential  Major  Work  Activities  -  This  is  a  taxonomy  of  work  activities  that 
will  help  you  to  derive  relevant  details  from  your  review  of  existing  information  about  your 
MOS. 

MOS  Characteristics  -  This  is  a  taxonomy  of  MOS  characteristics  that  can  affect  decisions 
about  the  types  of  measures  to  use  for  assessing  job  performance  in  a  particular  MOS. 

Step  2:  Review  Existing  MOS  Information 

Review  existing  information  about  your  MOS.  Likely  documents  include: 

•  Skill  Level  1  Soldier  Manuals  for  the  MOS.  These  can  be  very  detailed 
documents.  Try  not  to  be  overwhelmed  by  the  detail;  remember  you  are  aiming 
for  a  level  of  detail  similar  to  that  illustrated  in  the  example  form. 

•  A  task  list  from  the  Army’s  current  Army  Occupational  Survey  program.  Be 
careful;  they  vary  considerably  quality  for  our  purposes  and  how  up-to-date  they 
are. 

•  Manuals  for  particular  procedures  and/or  equipment.  This  can  be  valuable  and 
informative,  but  be  careful  not  to  get  too  far  into  the  weeds. 
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Notes:  Find  out  how  current  your  materials  are  and  whether  some  more  recent 
versions  are  about  to  be  published.  Also,  make  sure  to  keep  and  file  copies  of  these 
documents  because  more  careful  analysis  of  them  might  be  important  during  the 
detailed  job  analysis  and  measure  development  efforts. 

Step  3:  Complete  the  First  Draft  of  the  Preliminary  Job  Analysis  Form 

Review  of  existing  materials  should  provide  enough  information  for  a  first  draft  of  this  form. 

A.  Based  on  your  review  of  existing  materials  select  the  8  to  12  broad  activities  from  the 
Taxonomy  of  Potential  Major  Work  Activities  that  you  think  are  most  important  to 
successful  performance  in  this  MOS.  Record  the  titles  of  these  activities  in  your  form. 
Under  each  activity  include  the  best  two  to  five  examples  you  can  come  up  with  of 
MOS-specific  activities  that  illustrate  why  this  broad  activity  is  important  for  this 
MOS  (see  example  form). 

B.  Next,  evaluate  the  MOS  on  each  characteristic  in  the  list  of  MOS  Characteristics.  For 
each  of  the  Determinants  of  Performance  indicate  whether  the  MOS  appears  to 
require  a  high,  medium,  or  low  level  of  that  determinant  relative  to  other 
determinants.  (Note:  This  may  be  a  difficult  judgment  to  make,  but  do  your  best. 
There  will  be  opportunities  for  review  and  revision.).  For  those  determinants  that  you 
judge  to  be  required  at  a  medium  or  high  level,  provide  the  best  two  to  five  examples 
you  can  find  in  the  materials  of  MOS-specific  loiowledges  or  skills  that  illustrate  why 
this  general  knowledge  or  skill  area  is  important  for  this  MOS.  The  example  form 
indicates  that  1  IB  Infantryman  requires  a  High  level  of  the  Procedural  Knowledge  & 
Skill  (PK&S)  Manipulating  Objects  and  shows  the  following  illustrative  examples: 


■  PK&S  Manipulating  Objects  [High] 

-  Physical  Skills 

-  Skill  in  assembling  and  disassembling  a  personal  weapon 

-  Skill  in  navigating  with  a  ground  position  Location 
System  (GPS)  device 

-  Skill  in  administering  basic  first  aid 

-  Psychomotor  Skills 

-  Skill  in  firing  a  personal  weapon  at  a  moving  target 

_ -  Skill  in  zeroing  a  weapon _ 

C.  The  next  part  of  the  MOS  Characteristics  section  of  the  form  is  titled  Cognitive 
Complexity.  Here  you  are  asked  to  make  judgments  about  the  MOS’s  standing  on 
different  characteristics  that  can  result  in  it  demanding  more  or  less  cognitive 
capacity.  They  include  the  degree  of  Information  Intensity,  Judgment/Problem 
Solving,  and  Systems  Thinking  (see  MOS  Characteristics).  For  each,  indicate  the 
extent  to  which  you  think  the  characteristic  is  important  enough  to  performance  at 
Skill  Level  1  that  it  needs  to  be  assessed  by  the  job  performances  measure(s)  we  will 
develop  for  the  MOS.  Include  a  brief  narrative  supporting  your  judgment  (see 
example  form). 
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D.  Work  Context  comes  after  Cognitive  Complexity.  For  the  first  characteristic,  Method 
of  Getting  Information/Communicating,  just  record  your  best  judgment  about 
importance  of  each  of  the  four  methods  of  communication  to  Skill  Level  1 
performance  in  this  MOS.  For  Work  Context  characteristics  2  through  8,  indicate  the 
extent  to  which  you  think  the  characteristic  is  important  enough  to  performance  at 
Skill  Level  1  that  it  needs  to  be  assessed  by  the  job  performances  measures.  Include  a 
brief  narrative  supporting  your  judgment  (see  example  form). 

E.  The  next  section  is  Major  Tools,  Equipment,  and  Systems.  The  idea  here  is  to  list  the 
tools,  equipment  and/or  systems  that  you  think  are  important  to  the  MOS.  Examples 
include  personal  weapons,  navigation  equipment,  and  computer  systems  for 
communication  or  operating  crew  served  weapons  as  well  as  more  obvious  civilian 
equivalents  (e.g.,  large  truck  for  transporting  supplies,  construction  tools,  and  human 
resource  management  software).  An  important  caution  to  keep  in  mind  is  that  lists  of 
tools  and  equipment  for  many  MOS  are  over  inclusive.  That  is,  they  include  many 
items  that  are  used  by  only  of  minority  of  the  Skill  Level  1  Soldiers  in  a  particular 
MOS.  In  fact,  the  tools  and  equipment  used  in  some  MOS  are  so  assignment/unit 
specific  that  it  is  only  a  small  set  of  items  that  are  used  by  all  or  even  most  of  the 
individuals  in  an  MOS.  Even  at  this  early  stage,  do  your  best  to  stick  with  only  the 
tools,  equipment,  and  systems  that  are  most  relevant.  A  major  goal  in  latter  interviews 
and  reviews  will  be  to  identify  the  items  that  are  broadly  used  across  the  MOS  and/or 
that  need  to  be  tracked  in  the  performance  measures  (i.e.,  tools,  equipment,  and 
systems  that  are  very  important  to  some  in  the  MOS,  but  not  relevant  to  others).  The 
example  form  includes  examples  for  1  IB  Infantryman. 

F.  The  final  section  of  this  form  is  Practical  Considerations.  Here  you  should  include 
additional  information  that  you  think  is  relevant  to  developing  performance  measures 
for  this  MOS.  At  the  least,  you  should  include  a  brief  narrative  regarding  the  likely 
future  of  the  MOS  and  indicate  your  best  estimate  of  the  number  of  Skill  Level  1 
Soldiers  currently  in  the  MOS.  For  example,  some  MOS  are  in  the  process  of  being 
collapsed  into  others  or  are  so  small  that  it  might  be  not  very  beneficial  for  them  to  be 
part  of  the  initial  focus  of  this  effort  (see  example  form). 

Step  4:  Prepare  for  and  Conduct  Subject  Matter  Expert  (SME)  Interview  and  Job 
Observation 

The  objective  of  this  step  is  to  correct,  supplement,  add  to,  and  refine  your  first  draft  of  the 
Preliminary  Job  Analysis  Form. 

A.  Have  the  first  draft  reviewed  by  at  least  two  members  of  the  project  staff  who  are 
familiar  with  the  goals  of  the  preliminary  job  analysis.  Also,  to  the  extent  possible 
select  staff  members  who  are  familiar  with  the  MOS.  Ask  them  to  (a)  make  edits 
based  on  their  knowledge  of  the  MOS  and/or  the  Army  in  general  and  (b)  suggest 
questions  and  clarifications  to  focus  on  during  the  interview.  Additionally,  you  and 
the  reviewer  should  ask  yourselves  whether  there  are  any  outstanding  issues  that 
could  easily  be  resolved  with  another  source  of  information  (e.g.,  written  material  or 
an  SME  we  can  call)  before  the  formal  interview.  Make  any  necessary  revisions  at 
this  time. 
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B.  Develop  an  interview/job  observation  procedure  tailored  to  your  MOS.  The  first  part 
of  the  interview  should  include  a  standard  and  very  short  briefing  describing  the 
goals,  methods,  and  likely  outcomes  of  this  effort.  Next,  you  should  think  about  how 
to  explain  to  the  SME(s)  the  process  you  used  to  develop  the  first  draft  of  the  form 
and  ask  them  to  comment  on  each  part.  Use  the  form  itself  to  structure  the  interview. 
For  example,  explain  to  the  SME(s)  that  you  selected  the  8  to  12  broad  activities  that 
you  think  are  most  important  to  successful  performance  in  this  MOS.  Ask  them  to 
comment  on/revise  your  selections  and  give  you  feedback  on  your  illustrative  MOS- 
specific  activities.  Use  this  process  to  develop  a  procedure  for  reviewing  and  revising 
the  rest  of  the  form.  Here  are  some  likely  questions: 

•  Are  there  any  other  relevant  existing  materials  that  we  should  review? 

•  Is  this  activity,  determinant,  or  tool  important  for  everyone  in  this  MOS?  For  what 
units/assignments  is  it  most  important  and  for  what  proportion  of  the  existing 
Skill  Level  1  Soldiers  do  you  think  it  is  relevant? 

•  Did  I  select  the  correct  broad  activities  and  do  I  have  the  best  MOS-specific 
activities  as  examples? 

•  Did  I  judge  the  importance  of  each  determinant  correctly  and  did  I  pick  the  best 
example  knowledges  or  skills? 

•  Is  this  item  important  for  Skill  Level  1  Soldiers  or  does  it  apply  only  to  higher 
skill  levels? 

•  Are  likely  changes  in  this  MOS  going  to  make  this  item  more  or  less  important  in 
the  future? 

•  Is  this  item  important  enough  to  make  sure  it  is  included  in  an  assessment  of 
performance  for  this  MOS  when  you  consider  that  you  can’t  test  everything?  Here 
are  a  few  example  strategies  for  this  question: 

-  When  Soldiers  make  a  mistake  in  this  area  is  it  general  because  (a)  of  a 
lack  of  knowledge  about  what  to  do,  (b)  they  didn’t  get  enough  practice 
on  this  task,  or  (c)  they  just  weren’t  motivated  to  get  it  right? 

-  When  you  consider  all  the  things  that  a  fully  proficient  Skill  Level  1 
Soldier  should  be  able  to  do  would  you  give  this  item  a  high,  medium,  or 
low  priority  for  including  in  a  measure  of  performance? 

-  Does  competence  on  this  item  regularly  distinguish  good  performers 
from  poor  performers  in  this  MOS? 

•  During  the  preliminary  job  analysis,  job  observation  should  be  reserved  for  only 
those  occasions  when  it  seems  necessary  to  resolve  a  question  regarding  an 
element  of  information  on  the  form.  This  won’t  happen  often. 

C.  After  conducting  the  interview/job  observation,  prepare  notes  describing  all  the 
relevant  information  collected  and  develop  and  finalize  the  second  draft  of  the 
Preliminary  Job  Analysis  Form. 


45 


Step  5:  Prepare  Final  Version  of  the  Preliminary  Job  Analysis  Form 

Have  the  second  draft  reviewed  by  at  least  three  members  of  the  project  staff.  Make  sure  that 
at  least  one  of  them  was  a  reviewer  of  the  first  draft.  Finally,  prepare  the  final  version  of  the 
Preliminary  Job  Analysis  Form  for  your  MOS. 
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Example  Preliminary  Job  Analysis  Form 
11B  Infantryman  (Skill  Level  1) 

Major  MOS-Specific  Activities 

■  Performing  General  Physical  Activities 

-  Enter  buildings  during  an  urban  operation 

-  Install  pickets,  barbed  wire,  and  concertina 

■  Handling  and  Moving  Objects 

-  Construct  a  field-expedient  antenna 

-  Destroy  supplies  and  equipment 

■  Controlling  Machines  and  Processes 

-  Zero  a  weapon 

-  Engage  targets  with  a  weapon 

■  Communicating  with  Supervisors,  Peers,  or  Subordinates 

-  Use  visual  signaling  techniques  to  move  as  a  member  of  a  fire  team 

-  Report  position  to  supervisor  using  radio 


MOS  Characteristics 
Determinants  of  Performance 

■  Declarative  Knowledge  [Medium] 

-  Knowledge  of  steps  for  zeroing  a  weapon 

-  Knowledge  of  procedures  for  employing  aiming  devices  (e.g.,  night  and  thermal  sites  & 
aiming  lights) 

-  Knowledge  of  movement  techniques  during  urban  operations 

■  PK&S  Manipulating  Objects  [High] 

-  Physical  Skills 

-  Skill  in  assembling  and  disassembling  a  personal  weapon 

-  Skill  in  navigating  with  a  ground  position  Location  System  (GPS)  device 

-  Skill  in  administering  basic  first  aid 

-  Psychomotor  Skills 

-  Skill  in  firing  a  personal  weapon  at  a  moving  target 

-  Skill  in  zeroing  a  weapon 
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Cognitive  Complexity 

■  Information  Intensity  [Medium] 

-  Given  the  hierarchical  nature  of  infantry  fire  teams  and  squads,  the  amount  of 
information  an  11B  needs  to  process  at  once  is  generally  not  overwhelming  compared 
to  other  MOS. 

■  Judgment/Problem  Solving  [Low] 

-  Despite  the  fact  that  1  IBs  may  need  to  make  many  decisions,  at  Skill  Level  1  most 
situations  requiring  a  complex  decision  are  addressed  by  supervisors  or  covered  by  an 
order,  rule,  or  regulation. 


Work  Context 

■  Method  of  Getting  Information/Communicating 

-  Face-to-face  [High] 

-  Radio,  telephone  or  other  audio  device  [Medium] 

-  Manuals  or  other  non-computerized  text  [Low] 

-  Computer/Visual  Display  [Currently  Low/Likely  to  Increase  in  Future] 


■  Time  Pressure/Decision  Speed  [High] 

-  Given  the  nature  of  operating  as  a  member  of  a  fire  team  in  combat  and  training 
situations  the  necessity  to  make  decisions  quickly  is  high. 


Major  Tools,  Equipment,  and  Systems 

■  Individual  weapons  (i.e.,  M9,  M16  Series,  M203,  M240  Series,  M257,  MK19,  M249, 
M60,  .50  M2  Machine  Gun,  M242,  M4) 

■  Aiming  Devices 

-  Night  Sites 

-  Thermal  Sites 

-  Aiming  Lights 

■  Radios 

■  Compass 

■  Portable  Ground  Position  Location  System  (GPS)  device 
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Practical  Considerations 


■  This  MOS  is  likely  to  remain  in  its  current  (or  a  very  similar  form)  for  the  foreseeable 
future. 

■  This  MOS  is  very  populous.  It  has  sufficient  numbers  that  (a)  it  will  be  relatively 
straightforward  to  gain  access  to  a  sufficient  number  of  subject  matter  experts  and 
Soldiers  to  develop  measures  and  (b)  an  effective  measure  of  job  performance  for  llBs 
would  affect  a  substantial  portion  of  the  Army’s  first  tour  population. 
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Taxonomy  of  Potential  Major  Work  Activities 
(Adapted  From  0*NET  Generalized  Work  Activities) 

1 .  Getting  Information  -  Observing,  receiving,  and  otherwise  obtaining  information  from  relevant 
sources. 

2.  Identifying  Objects,  Actions,  and  Events  -  Identifying  information  by  categorizing,  estimating, 
recognizing  differences  or  similarities,  and  detecting  changes  in  circumstances  or  events. 

3.  Monitoring  Processes,  Materials,  or  Surroundings  —  Monitoring  and  reviewing  information  from 
materials,  events,  or  the  environment,  to  detect  or  assess  problems. 

4.  Inspecting  Equipment,  Structures,  or  Materials  -  Inspecting  equipment,  structures,  or  materials  to 
identify  the  cause  of  errors  or  other  problems  or  defects. 

5 .  Estimating  the  Quantifiable  Characteristics  of  Products,  Events,  or  Information  -  Estimating 
sizes,  distances,  and  quantities;  or  determining  time,  resources,  or  materials  needed  to  perform  a  work 
activity. 

6.  Judging  the  Qualities  of  Objects,  Services,  or  People  -  Assessing  the  value,  importance,  or  quality 
of  things  or  people. 

7.  Evaluating  Information  to  Determine  Compliance  with  Standards  -  Using  relevant  information 
and  individual  judgment  to  determine  whether  events  or  processes  comply  with  regulations,  polices  or 
procedures. 

8.  Processing  Information  -  Compiling,  coding,  categorizing,  calculating,  tabulating,  auditing,  or 
verifying  information  or  data. 

9.  Analyzing  Data  or  Information  -  Identifying  the  underlying  principles,  reasons,  or  facts  of 
information  by  breaking  down  information  or  data  into  separate  parts. 

10.  Making  Decisions  and  Solving  Problems  -  Analyzing  information  and  evaluating  results  to  choose 
the  best  solution  and  solve  problems  in  situations  where  judgment  is  required  (i.e.,  when  the  problem 
cannot  be  straightforwardly  resolved  by  applying  a  specific  knowledge[s]  or  skilljs]). 

11.  Thinking  Creatively  -  Developing,  designing,  or  creating  new  applications,  ideas,  relationships, 
systems,  or  products,  including  artistic  contributions. 

12.  Updating  and  Using  Relevant  Knowledge  -  Keeping  up-to-date  technically  and  applying  new 
knowledge  to  your  job. 

13.  Scheduling  Activities  -  Scheduling  events,  programs,  and  activities. 

14.  Organizing,  Planning,  and  Prioritizing  Work  —  Developing  specific  goals  and  plans  to  prioritize, 
organize,  and  accomplish  your  work. 

15.  Performing  General  Physical  Activities  -  Performing  physical  activities  that  require  considerable 
use  of  your  arms  and  legs  and  moving  your  whole  body,  such  as  climbing,  lifting,  balancing,  walking, 
stooping,  and  handling  of  materials. 

16.  Handling  and  Moving  Objects  -  Using  hands  and  arms  in  handling,  installing,  positioning,  and 
moving  materials,  and  manipulating  things. 

17.  Controlling  Machines  and  Processes  -  Using  either  control  mechanisms  or  direct  physical  activity 
to  operate  machines  or  processes  (not  including  computers  or  vehicles). 

18.  Working  with  Computers  -  Using  computers,  computer  systems,  or  other  computer  based 
technology  (including  hardware  and  software)  to  program,  write  software,  set  up  functions,  enter  data, 
or  process  information. 

19.  Operating  Vehicles,  Mechanized  Devices,  or  Equipment  -  Running,  maneuvering,  navigating,  or 
driving  vehicles  or  mechanized  equipment,  such  as  forklifts,  passenger  vehicles,  infantry  fighting 
vehicles,  tanks,  or  water  craft. 

20.  Drafting,  Laying  Out,  and  Specifying  Technical  Devices,  Parts,  and  Equipment  -  Providing 
documentation,  detailed  instructions,  drawings,  or  specifications  to  tell  others  about  how  devices, 
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parts,  equipment,  or  structures  are  to  be  fabricated,  constructed,  assembled,  modified,  maintained,  or 
used. 

21.  Repairing  and  Maintaining  Mechanical  Equipment  -  Servicing,  repairing,  adjusting,  and  testing 
machines,  devices,  moving  parts,  and  equipment  that  operate  primarily  on  the  basis  of  mechanical 
(not  electronic)  principles. 

22.  Repairing  and  Maintaining  Electronic  Equipment  -  Servicing,  repairing,  calibrating,  regulating, 
fine-tuning,  or  testing  machines,  devices,  and  equipment  that  operate  primarily  on  the  basis  of 
electrical  or  electronic  (not  mechanical)  principles. 

23.  Documenting/Recording  Information  -  Entering,  transcribing,  recording,  storing,  of  maintaining 
information  in  written  or  electronic/magnetic  form. 

24.  Interpreting  the  Meaning  of  Information  for  Others  -  Translating  or  explaining  what  information 
means  and  how  it  can  be  used. 

25.  Communicating  with  Supervisors,  Peers,  or  Subordinates  -  Providing  information  to  supervisors, 
coworkers,  and  subordinates  by  telephone,  in  written  form,  e-mail  (or  other  electronic  devices),  or  in 
person. 

26.  Communicating  with  People  Outside  the  Organization  —  Communicating  with  people  outside  the 
organization,  representing  the  organization  to  the  public,  other  parts  of  the  military  or  government, 
and/or  indigenous  personnel.  The  information  can  be  exchanged  in  person,  in  writing,  or  by  telephone 
or  e-mail  (or  other  electronic  devices). 

27.  Establishing  and  Maintaining  Interpersonal  Relationships  -  Developing  constructive  and 
cooperative  working  relationships  with  others,  and  maintaining  them  over  time. 

28.  Assisting  and  Caring  for  Others  -  Providing  personal  assistance,  medical  attention,  emotional 
support,  or  other  personal  care  to  others  such  as  coworkers,  indigenous  personnel,  patients,  or  clients. 

29.  Influencing  Others  -  Convincing  others  to  change  their  minds  or  actions. 

30.  Resolving  Conflicts  and  Negotiating  with  Others  -  Handling  complaints,  settling  disputes,  and 
resolving  grievances  and  conflicts,  or  otherwise  negotiating  with  others. 

31.  Working  Directly  with  Non-Military  Personnel  -  Dealing  directly  with  the  public.  This  includes 
serving  or  directing  military  dependents,  other  U.S.  citizens,  foreign  military  personnel,  prisoners  of 
war,  and  foreign  indigenous  civilians. 

32.  Contributing  to  and  Supporting  Teams  -  Providing  assistance  and  support  to  team  members  and 
helping  the  team  remain  focused  on  its  goals. 

33.  Performing  Administrative  Activities  -  Performing  day-to-day  administrative  tasks  such  as 
maintaining  information  files  and  processing  paperwork. 

34.  Monitoring  and  Controlling  Resources  -  Monitoring  and  controlling  the  expenditure  of  resources. 
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Definitions  of  MOS  Characteristics 


Determinants  of  Performance 

1.  Declarative  Knowledge  -  knowledge  of  facts  and  things;  including  lists  of  steps  in  a 
procedure  (i.e.,  knowing  what  to  do). 

Note:  In  very  practical  terms  this  means  knowledge  (including  some  knowledge 
application)  that  could  be  assessed  with  a  multiple-choice  test,  including  test  items  with 
graphical,  audio,  and/or  visual  stems  and/or  response  options. 

2.  Procedural  Knowledge  &  Skill  (PK&S)/Manipulating  Objects  -  (e.g.,  assembling  a  weapon, 
operating  a  cash-register,  aiming  a  large  weapon) 

-  Physical  Skills  -  performing  activities  that  require  strength,  endurance,  flexibility, 
balance,  and/or  coordination 

-  Psychomotor  Skills  -  gross  or  fine  manipulation  of  objects  requiring  precision  and/or 
control  (e.g.,  rate  control,  response  orientation,  reaction  time,  speed  of  movement) 

Note:  The  following  definition  applies  to  all  of  the  PK&  S  characteristics.  Procedural 
Knowledge  &  Skill  -  the  successful  combination  of  “knowing  what  to  do”  with 
“knowing  how  to  do  it;”  this  is  especially  prominent  when  mastery  depends  on  practice 
(e.g.,  word  processing  or  interacting  with  customers).  Test  items/scenarios  addressing 
procedural  knowledge  focus  on  requiring  incumbents  to  simulate  doing  part  of  their 
work. 

3.  PK&S/Interacting  with  others  -  effective  interaction  with  others  (e.g.,  team  members, 
supervisors,  civilians,  indigenous  military  personnel) 

-  Interpersonal  Skills 

-  Listening 

-  Speaking 

4.  PK&S/Using  information  resources  -  Finding  information  in  a  manuals,  data  bases,  or  on  a 
intra/intemet 

5.  PK&S/Reading 

6.  PK&S/Writing  [Reading  and  writing  refer  specifically  to  activities  required  by  the  MOS,  not 
to  these  skills  in  the  general  context.] 

Cognitive  Complexity 

1.  Information  Intensity  -  complex  decision-making  and/or  processing  a  lot  of  information  at 
once. 

2.  Judgment/Problem  Solving  -  situations  where  the  Soldier  needs  to  exercise  judgment 
regarding  problems  that  are  not  easily  addressed  by  a  specific  knowledge(s)  or  skill(s).  That 
is,  situations  in  which  judgment  might  be  important,  even  for  Skill  Level  1  Soldiers:  (a)  there 
is  a  regulation,  rule,  or  procedure,  but  recognizing  that  it  is  relevant  or  necessary  to  follow  is 
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difficult  for  some;  (b)  the  manuals  don’t  cover  this  (a  lot  of  interpersonal,  team,  and 
leadership  stuff  might  fall  in  this  category);  and  (c)  different,  potentially  contradictory, 
policies  seem  to  be  relevant  to  the  situation. 

3.  Systems  Thinking  -  understanding  a  “system”  with  a  number  of  interrelated  elements  that 
affect  each  other.  Examples  of  jobs  high  on  this  characteristic  include,  Civilian  Air  Traffic 
Controller,  Water  Treatment  Specialist,  and  Dispatcher.  An  MOS  completely  lacking  in  this 
characteristic  would  be  one  in  which  very  few  of  its  tasks  are  dependent  on  each  other  (i.e.,  a 
number  of  discrete  task  where  performance  on  one  tasks  is  not  affected  by  performance  on 
the  others). 

Work  Context 

1.  Method  of  Getting  Information  and  Communicating  with  Others  -  How  does  the  solider  get 
the  information  necessary  to  perform? 

-  Computer/Visual  Display 

-  Radio,  telephone,  or  other  audio  device 

-  Manuals,  memos,  or  other  non-computerized  text 

-  Face-to-face  personal  communication 

2.  Level  of  Social  Interaction  -  amount  and  complexity  of  interpersonal  contact  with  others 

3.  Level  of  Interpersonal  Conflict/Strained  Interaction  -  extent  to  which  the  MOS  requires  or 
results  in  conflict  with  others  or  dealing  with  individuals  who  are  angry  or  hostile 

4.  Exposure  to  Extreme  Environmental  Conditions  or  Hazards  -  e.g.,  temperature,  noise,  light, 
air  contaminants,  CBRN  hazards,  and/or  other  situations  requiring  safety  equipment, 
clothing,  or  procedures 

5.  Level  of  Vigilance  -  extent  to  which  the  MOS  requires  the  Soldier  to  maintain  attention  or 
alertness,  either  for  events  or  circumstances  that  do  not  occur  often  or  for  those  that  are 
subject  to  continual  change 

6.  Time  pressure/decision  speed  -  performing  work  or  making  decisions  quickly 

7.  Attention  to  Detail  -  extent  to  which  the  MOS  requires  thoroughness  to  make  sure  nothing  is 
left  undone  or  that  all  steps  are  taken  in  the  proper  order 

8.  Distractions  and  Interruptions  -  extent  to  which  the  Soldier  can  expect  to  distractions  and/or 
interruptions  and  the  extent  to  which  these  are  under  the  Soldiers  control 


Major  Tools,  Equipment,  and  Systems 

A  list  of  the  major  tools,  pieces  of  equipment,  and  systems  that  are  used  by  all  or  most  of  the 
Skill  Level  1  Soldiers  in  this  MOS. 
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Appendix  F 

Assessment  Design  Guide  Template 
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ASSESSMENT  DESIGN  GUIDE 

TEMPLATE 


<INSERT  MOS  TITLE> 

SKILL  LEVEL  <1  OR  2> 


SBIR ,  A 2 -  1  2 2 2 ,  PHASE  II 

THE  U.S.  ARMY  RESEARCH  INSTITUTE  FOR  THE 
BEHAVIORAL  AND  SOCIAL  SCIENCES 


<INSERT  DATE> 
<INSERT  REVISION  NUMBER> 


Section  I  -  Background 

Job  title  (Skill  level): 

dnsert  job  title  and,  in  parentheses,  targeted  skill  level  i.e.,  1  or  2> 


MOS  Description: 

dnsert  brief  description  of  MOS. > 

Start  with  summary  of  Major  Duties.  Follow  by  description  of  major  tasks/activities. 
Description  can  be  taken  from  the  set  of  descriptions  of  Army  MOS  derived  from  U.S.  Army 
Pamphlet  611-21  (Department  of  the  Army,  1999)  and  used  in  the  Phase  I SBIR  effort. 

Assessment  Strategy 

Describe  the  measurement  method  (or  methods)  to  be  used.  Select  from  the  list  shown  below. 
Provide  any  information  available  about  desired  length  of  the  assessment  (e.g.,  number  of  test 
items,  time  to  complete).  Indicate  means  by  which  the  measure  will  be  administered  and  scored. 


Assessment  Method  Options 

Computer-Based  Testing 

Independent  multiple-choice  items 

Multiple-choice  items  presented  as  part  of  job  scenarios 

Low  to  medium  fidelity  interactive  simulation 

High  fidelity  interactive  simulation 

Situational  judgment  test  items 

Actual  Job  Performance 

Hands  on  demonstration 

Historical  records 

Embedded  on-the-job  monitoring 

Evaluation  of  final  work  products 

Section  II  -  Assessment  Overview 

Using  the  heading  shown  below,  provide  an  overview  of  the  design  of  the  measure  and  how  it 
will  function.  Describe  the  elements  of  the  assessment  and  how  they  will  function.  Indicate  how 
the  user  will  interact  with  these  elements.  As  appropriate,  prepare  a  structure  diagram  for 
computer-based  tests  that  clarify  the  possible  paths  a  user  can  take  to  proceed  through  the  testing 
activities.  Describe  actions  the  user  takes  to  physically  navigate  from  screen  to  screen.  Describe 
how  the  user  will  be  instructed  in  the  use  of  the  assessment. 
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Assessment  Elements  and  Functionality 


Structure 

Navigation 

Instructions 

Section  III  -  Deliverables 

List  and  describe  all  project  deliverables  (draft  and  final).  Indicate  number  of  copies  of  each  to 
be  delivered.  Indicate  which  products  are  to  be  delivered  in  hard  copy  and  which  in  electronic 
form.  For  software,  indicate  what  source  code  and  what  executables  will  be  provided.  Indicate 
what  software  documentation  will  be  provided. 

Section  IV  -  System  Requirements 

Indicate  all  software  and  hardware  requirements  that  will  be  needed  to  operate  the  measures. 

PC  Computer  System  (check  all  that  apply) 

_ 486 

_ Pentium 

_ Pentium  2 

_ Other _ 

Macintosh  compatible  Yes _ No _ 

CPU 

Minimum  recommended  processor  speed _ 

Memory  (indicate  amounts) 

Minimum  RAM _  Recommended _ 

Minimum  hard  drive  space _  Recommended _ 

Operating  System 

Windows  3.1  Yes _  No _ 

Windows  95  Yes _  No _ 

Windows  98  Yes _  No _ 

Windows  2000  Yes _  No _ 

Windows  XP  Yes _ No _ 

MAC  O/S  Yes _  No _  If  Yes,  then  version _ 

UNIX  Yes _  No _ 

Windows  NT  Workstation  Yes _  No _ 

Multimedia  Components 

Sound  card  with  speakers  Yes _  No _ 

Sound  card  type  (e.g.,  Sound  Blaster  compatible) _ 

CD-ROM:  Yes _  No _ 

Other _ _ _ 
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Software  Plug-ins 

Flash  Player  Yes _  No _ 

Other  (list) : _ _ _ 

Communications 

Network/Intemet  Yes _  No _ 

Recommended  Baud _ 

Monitor 

Size  of  screen  (e.g.,  15  inch,  17  inch,  19  inch) _ 

Recommended  resolution  (e.g.,  800  by  600  or  1024  by  768) - 

Number  of  colors  (e.g.,  256 ) _ 

Browser 

Netscape  Yes _ No _  (if  yes  then  minimum  version _ ) 

Explorer  Yes _ No _  (if  yes  then  minimum  version _ ) 

Other  _ _ _ 

Other  (e.g.,  peripherals): _ _ _ 


Section  V  -  Project  Structure  and  Management 

Provide  a  breakdown  of  the  project  into  major  tasks  and  subtasks.  For  major  tasks,  select  from 
the  list  below: 

•  Preliminary  Job  Analysis 

•  Formal  Job  Analysis 

•  Assessment  Design 

•  Measure  Development 

•  Pilot  Testing 

•  Final  Revisions 

•  Report  Writing 

•  Software  Documentation 

•  Project  Management  Activities 

Generate  subtasks  as  needed.  Enter  the  structure  into  the  JPS  automated  time  keeping  system 
(HumRRO  will  use  a  comparable  paper  based  system).  This  information  will  be  used  to  manage 
the  project  and,  at  the  conclusion  of  the  project,  as  metrics  to  help  identify/evaluate  process 
improvement  efforts. 

Select  project  roles  (e.g.,  project  manager,  computer  programmer,  writer/editor,  instructional 
designer)  from  the  automated  time  keeping  system.  Associate  individual  employees  to  these 
roles  and  provide  permissions  for  them  to  charge  to  appropriate  tasks/subtasks.  Coordinate  with 
HumRRO  to  ensure  we  can  obtain  comparable  data. 
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The  project  manager  is  responsible  for  preparing  a  project  budget/planning  spreadsheet.  This 
person  must  update  the  budget  on  a  monthly  basis.  This  person  must  also  update  the  company 
master  JPS  coverage  spreadsheet  at  the  start  of  each  month  so  JPS  management  can  use  this  tool 
to  help  ensure  all  projects  have  sufficient  resources. 

Section  VI  -  Project  Schedule 

<insert  project  schedule  hero 

Describe  any  significant  risks  you  foresee  in  meeting  the  schedule  (e.g.,  delayed  start  date,  lack 
of  SME  support,  use  of  new  programming  tools)  and  actions  that  need  to  happen  to  address  these 
risks.  Complete  a  table  such  as  the  one  below  to  indicate  dates  when  client  is  to  receive 
materials  for  review  and  when  the  reviews  must  be  completed. 


Client  Reviews 


Item 

Dates  (When  Provided  When  to 

Be  Returned) 

Comments 

Complete  a  table  such  as  the  one  below  to  clarify  what  SMEs  will  be  required  and  when  they  be 
needed. 


SME  Requirements 


Date 

SMEs 

Purpose 

Section  VII  -  Communications 
Formal 

JPS  will  submit  monthly  status  reports  according  to  the  requirements  stated  in  our  contract  with 
the  U.S.  Army.  The  report  will  describe  work  completed  during  the  reporting  period,  work  to  be 
completed  during  the  next  period,  and  any  issues. 
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Informal 


JPS  will  communicate  by  phone  and  email  with  the  client  on  a  routine  basis,  as  needed, 
throughout  the  life  of  the  project. 

Section  VIII  -  Page  Designs 

General  Considerations 

Obtain  from  sponsor  any  Army  or  Army  proponent  requirements  for  items  on  screens  (e.g., 
logos,  symbols,  colors,  headers,  footers).  Describe  the  nature  of  any  photos,  videos,  graphics, 
etc.  you  anticipate  adding  to  screens.  Determine  who  can  provide  such  information  and  what 
releases  may  be  needed  before  they  can  be  used. 

Depending  on  the  design  of  the  assessment,  indicate  the  main  color  scheme  to  be  used  (see 
example  shown  below).  Minimum  palette  colors  typically  consist  of  a  primary  color,  an  accent 
color,  and  a  secondary  color.  The  primary  and  secondary  web-safe  colors  should  provide  an 
easy-to-view  contrast  against  the  interface  background.  For  web  delivered  products  select  accent 
colors  that  are  web  safe. 


003399 


Secondary 

FFS600 


Primary 

Haettenschweller 


Sample  Screen  Pages 

Use  this  section  to  provide  sample  screen  pages.  For  any  computer-based  test  you  will  likely 
need  to  provide,  at  a  minimum,  the  following  samples: 

•  Splash  page 

•  Registration  page 

•  Instructions  page 

•  Simulated  equipment,  tools,  job  aids 

•  Other  items  of  job  information 

•  Test  items 

•  End-of-test  page 
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Section  IX  -  Assessment  Content  and  Sequence 

Typically  one  of  the  early  steps  in  the  design  phase  is  to  produce  scripts  and/or  storyboards  of 
the  content.  These  items  indicate  the  sequence  of  events,  the  text  to  be  displayed  on  the  screen, 
interactivity,  and  any  branching. 

For  the  design  guide,  indicate  how  the  content  and  sequence  will  be  portrayed  (e.g.,  MS  Word 
document  listing  individual  test  items,  scripts,  storyboards).  Provide  as  much  information  as 
possible  at  this  stage  regarding  the  number,  length,  detail  required  in  these  documents.  If 
possible  provide  a  template  for  these  documents. 


Section  X  -  Quality  Control 

Indicate  the  timing  when  formal  quality  control  checks  will  be  made  on  products.  Indicate  which 
checks  will  be  conducted  by  members  of  the  JPS  team  and  by  SMEs.  Describe  the  goals  of  the 
pilot  test  and  how  we  will  determine  information  we  need  to  ensure  the  quality  of  the  measures. 

Section  XI  -  Process  Improvement 

One  of  our  goals  in  producing  any  performance  assessment  is  make  demonstrated  progress  in 
improving  the  efficiency  of  the  process  for  the  next  assessment.  There  are  two  types  of  process 
improvement  efforts.  One  is  designed  to  reduce  time/costs.  The  other  is  to  maintain  or  improve 
quality.  In  this  section,  indicate  the  strategy  for  achieve  improvements  in  both  areas. 

Indicate  what  new  tools,  procedures,  or  templates  will  be  developed  to  make  the  process  we 
follow  more  standardized  and  streamlined.  Select  one  or  more  test  development  steps  for  which 
we  can  reduce  the  time  and  effort  required  without  a  reduction  in  the  quality  of  the  output.  Use 
historical  data  to  set  targets  for  a  reduction  in  time/costs.  For  each  process  improvement 
initiative,  determine  the  quantitative  and  qualitative  metrics  that  will  be  tracked  to  support  these 
efforts. 

Section  XII  -  Assumptions 

Summarize  any  key  assumptions  you  are  making  regarding  this  project. 
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Appendix  G 

Assessment  of  Realism  in  Computer-Based  Tests 
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Part  1:  Evaluation  of  the  Design  and  Operation  of  the  Assessment 

Instructions:  Individuals  who  complete  Part  1  should  have  in-depth  knowledge  of  the  job.  They  should 
also  have  completed  the  assessment  or  spent  sufficient  time  observing  the  test  to  fully  understand  its 
content  and  operation. 

Physical  Fidelity 

1.  Stimuli  Characteristics 

Rate  the  extent  to  which  test  information  is  presented  in  a  manner  equivalent  to  how  the  same 
information  is  provided  on  the  job.  Consider  if  the  mode  (i.e.,  text,  pictures,  auditory,  tactile)  is 
the  same.  Then  consider  how  closely  the  presentation  of  information  between  the  job  and  test 
match. 

_ Low  (For  almost  all  of  the  important  stimuli  the  modes  do  not  even  match) 


Moderate  (The  modes  match  but  the  stimuli  are  only  somewhat  similar) 


_ High  (For  almost  all  of  the  important  stimuli  there  is  a  very  high  match) 

2.  Response  Characteristics 

Rate  the  extent  to  which  the  assessment  provides  response  options  equivalent  to  those  available 
on  the  job.  Consider  if  the  method  for  responding  is  equivalent  (i.e.,  writing,  typing,  speaking, 
psychomotor  action).  Then  consider  how  closely  the  test  responses  truly  mimic  the  same 
responses  on  the  job. 

_ Low  (The  method  of  responses  do  not  match) 


Moderate  (The  methods  match  but  the  manner  of  making  responses  are  different) 


.High  (Responding  is  virtually  identical  between  the  assessment  and  the  job) 


3.  Contextual  Factors 

Consider  the  extent  to  which  such  factors  as  background  noise,  feeling  of  motion,  light 
conditions,  distractions,  sounds  made  by  equipment  one  uses,  and  other  significant  contextual 
factors  are  present.  Then  consider  how  closely  these  factors  are  duplicated  by  the  assessment. 

_ Low  (No  contextual  factors  are  present,  or  they  bear  little  resemblance  to  the  job) 


.Moderate  (Contextual  factors  are  present  but  only  of  moderate  similarity) 


.High  (Factors  are  present  and  of  high  fidelity) 
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Psychological  Fidelity 

4.  Relevance  of  the  Content 

To  what  extent  do  the  tasks,  challenges,  situations,  reflect  important  components  of  job 
performance? 

_ Low  (Test  content  reflects  events  that  are  unimportant  or  unlikely  to  occur) 

_ Moderate  (Content  reflects  activities  of  moderate  important) 

_ High  (Content  focuses  completely  on  important  aspects  of  the  job) 

5.  Quality  of  the  Test  Content 

To  what  extent  is  the  content  well  constructed?  To  what  extent  is  it  accurate,  sequenced 
appropriately,  and  sufficiently  complete  to  be  true  to  the  job? 

_ Low  (Content  is  poorly  constructed) 

_ Moderate  (Content  is  somewhat  well  constructed) 

_ High  (Content  is  well  constructed) 

Additional  Characteristics  that  Can  Influence  Realism 

6.  Timing  and  Pace 

To  what  extent  is  the  timing  and  pacing  of  the  test  appropriate?  The  pace  should  not  be  slower 
than  one  would  expect  to  find  on  the  job.  It  should  not  be  so  fast  that  it  creates  a  unrealistic 
speeded  test. 

Low  (Pacing  is  much  too  slow  or  fast) 

_ Moderate  (Pacing  is  somewhat  too  slow  or  fast) 

_ High  (Pacing  is  highly  appropriate) 
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7.  Span  of  Events 

Indicate  the  extent  to  which  the  events  covered  in  the  test  span  a  significant  portion  of  the  job 
(e.g.,  large  tasks  or  activities). 

Low  (The  test  measures  a  number  of  unrelated  actions) 

_ Moderate  (The  test  contains  events  that  are  somewhat  well  related  and  sequenced) 

High  (The  test  effectively  integrates  events  across  large  tasks  and  job  activities) 

Features  Necessary  to  Support  Test  Validity 

8.  Representativeness 

To  what  degree  does  the  content  reflect  a  representative  sample  of  the  important  behaviors  in  the 
job? 

Low  (The  test  focuses  on  just  a  small  portion  of  the  important  tasks) 

_ Moderate  (The  test  focuses  on  about  half  of  the  important  tasks  in  the  job) 

_ High  (The  test  samples  from  all  important  tasks  in  the  job) 

9.  Meaningfulness  of  Scores 

Evaluate  the  quality  of  the  logic  used  in  scoring  test  performance,  combining  scores,  and  setting 
a  level  of  acceptable  performance. 

_ Low  (No  convincing  rationale  exists  to  explain  the  scoring  system) 

_ Moderate  (Some  logic  exists) 

_ High  (The  rationale  is  clear  and  convincing) 

10.  Standardization 

To  what  extent  is  it  likely  the  test  will  be  administered  in  a  standardized  manner  each  time. 

_ Low  (Testing  depends  heavily  on  the  administrator.  Any  automation  is  unreliable) 

_ Moderate  (The  majority  of  the  time  the  testing  will  be  standardized) 

High  (All  testing  is  automated  and  reliable.  The  administrator’s  role  is  minimal) 
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Part  2:  Reactions  of  the  Test  Taker 

Instructions:  Individuals  who  complete  Part  2  should  be  Soldiers  with  substantial  experience  in  the  job. 
They  should  complete  this  section  of  the  questionnaire  upon  finishing  the  test. 

Please  indicate  the  extent  to  which  you  disagree  or  agree  with  the  following  statements: 

1 .  In  taking  this  test,  I  made  the  same  kinds  of  judgments  that  I  make  on  the  job. 

_ Strongly  Disagree 

_ Disagree 

_ Neither  Agree  no  Disagree 

_ Agree 

_ Strongly  Agree 

2.  The  information  displayed  in  the  test  was  very  realistic. 

_ Strongly  Disagree 

_ Disagree 

_ Neither  Agree  no  Disagree 

_ Agree 

_ Strongly  Agree 

3.  The  test  provides  the  same  look  and  feel  as  the  real  job. 

_ Strongly  Disagree 

_ Disagree 

_ Neither  Agree  no  Disagree 

_ Agree 

_ Strongly  Agree 

4.  This  was  an  engaging  test  to  take. 

_ Strongly  Disagree 

_ Disagree 

_ Neither  Agree  no  Disagree 

_ Agree 

_ Strongly  Agree 

5.  The  test  provoked  the  same  kind  of  emotional  responses  in  me  that  I  experience  when 
performing  these  tasks  on  the  job. 

_ Strongly  Disagree 

_ Disagree 

_ Neither  Agree  no  Disagree 

_ Agree 

_ Strongly  Agree 
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